---
title: "serverless-spark-create-pyspark-batch"
type: docs
weight: 2
description: >
  A "serverless-spark-create-pyspark-batch" tool submits a Spark batch to run asynchronously.
---

## About

A `serverless-spark-create-pyspark-batch` tool submits a Spark batch to a Google
Cloud Serverless for Apache Spark source. The workload executes asynchronously
and takes around a minute to begin executing; status can be polled using the
[get batch](serverless-spark-get-batch.md) tool.

`serverless-spark-create-pyspark-batch` accepts the following parameters:

- **`mainFile`**: The path to the main Python file, as a gs://... URI.
- **`args`** Optional. A list of arguments passed to the main file.
- **`version`** Optional. The Serverless [runtime
  version](https://docs.cloud.google.com/dataproc-serverless/docs/concepts/versions/dataproc-serverless-versions)
  to execute with.


## Compatible Sources

{{< compatible-sources >}}

## Example

```yaml
kind: tool
name: serverless-spark-create-pyspark-batch
type: serverless-spark-create-pyspark-batch
source: "my-serverless-spark-source"
runtimeConfig:
  properties:
    spark.driver.memory: "1024m"
environmentConfig:
  executionConfig:
    networkUri: "my-network"
```

### Custom Configuration

This tool supports custom
[`runtimeConfig`](https://docs.cloud.google.com/dataproc-serverless/docs/reference/rest/v1/RuntimeConfig)
and
[`environmentConfig`](https://docs.cloud.google.com/dataproc-serverless/docs/reference/rest/v1/EnvironmentConfig)
settings, which can be specified in a `tools.yaml` file. These configurations
are parsed as YAML and passed to the Dataproc API.

**Note:** If your project requires custom runtime or environment configuration,
you must write a custom `tools.yaml`, you cannot use the `serverless-spark`
prebuilt config.

## Output Format

The response contains the
[operation](https://docs.cloud.google.com/dataproc-serverless/docs/reference/rest/v1/projects.locations.operations#resource:-operation)
metadata JSON object corresponding to [batch operation
metadata](https://pkg.go.dev/cloud.google.com/go/dataproc/v2/apiv1/dataprocpb#BatchOperationMetadata),
plus additional fields `consoleUrl` and `logsUrl` where a human can go for more
detailed information.

```json
{
  "opMetadata": {
    "batch": "projects/myproject/locations/us-central1/batches/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
    "batchUuid": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
    "createTime": "2025-11-19T16:36:47.607119Z",
    "description": "Batch",
    "labels": {
      "goog-dataproc-batch-uuid": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
      "goog-dataproc-location": "us-central1"
    },
    "operationType": "BATCH",
    "warnings": [
      "No runtime version specified. Using the default runtime version."
    ]
  },
  "consoleUrl": "https://console.cloud.google.com/dataproc/batches/...",
  "logsUrl": "https://console.cloud.google.com/logs/viewer?..."
}
```

## Reference

| **field**         | **type** | **required** | **description**                                                                                                                                          |
| ----------------- | :------: | :----------: | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| type              |  string  |     true     | Must be "serverless-spark-create-pyspark-batch".                                                                                                         |
| source            |  string  |     true     | Name of the source the tool should use.                                                                                                                  |
| description       |  string  |    false     | Description of the tool that is passed to the LLM.                                                                                                       |
| runtimeConfig     |   map    |    false     | [Runtime config](https://docs.cloud.google.com/dataproc-serverless/docs/reference/rest/v1/RuntimeConfig) for all batches created with this tool.         |
| environmentConfig |   map    |    false     | [Environment config](https://docs.cloud.google.com/dataproc-serverless/docs/reference/rest/v1/EnvironmentConfig) for all batches created with this tool. |
| authRequired      | string[] |    false     | List of auth services required to invoke this tool.                                                                                                      |
