DevOps

AWS Glue BigQuery Connection Go SDK v2: Fix SparkProperties & secretId

Fix InvalidInputException when creating AWS Glue native BigQuery connection with Go SDK v2. Configure SparkProperties JSON with secretId for Secrets Manager GCP credentials correctly.

1 answer• 1 view

AWS Glue native BigQuery connection creation fails with Go SDK v2: “SparkProperties is missing but it is required” and “secretId is not defined in the schema”

I’m attempting to programmatically create a native Google BigQuery connection in AWS Glue using the AWS SDK for Go v2 (github.com/aws/aws-sdk-go-v2/service/glue).

Per AWS documentation (Glue 4.0+ supports native BigQuery connections), the minimal setup requires:

  • ConnectionType: BIGQUERY
  • SECRET_ID in ConnectionProperties
  • No SparkProperties (used for custom/marketplace connectors)

However, every attempt results in an InvalidInputException:

InvalidInputException: secretId: is not defined in the schema and the schema does not allow additional properties, SparkProperties: is missing but it is required

Here’s the code snippet I’m using:

go
import (
 "context"
 "fmt"

 "github.com/aws/aws-sdk-go-v2/aws"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/service/glue"
 gluetypes "github.com/aws/aws-sdk-go-v2/service/glue/types"
)

func main() {
 cfg, _ := config.LoadDefaultConfig(context.TODO())
 client := glue.NewFromConfig(cfg)

 secretName := "my-bq-secret" // secret name

 input := &glue.CreateConnectionInput{
 ConnectionInput: &gluetypes.ConnectionInput{
 Name: aws.String("test-bq-native-conn"),
 Description: aws.String("Test native BigQuery connection"),
 ConnectionType: gluetypes.ConnectionTypeBigquery,
 ConnectionProperties: map[string]string{
 "SECRET_ID": secretName,
 },
 // SparkProperties omitted 
 // PhysicalConnectionRequirements omitted for minimal test
 },
 }

 _, err := client.CreateConnection(context.TODO(), input)
 if err != nil {
 fmt.Printf("Error: %v\n", err)
 } else {
 fmt.Println("Success")
 }
}

How do I correctly configure the ConnectionInput for a native BigQuery connection in AWS Glue using Go SDK v2? What are the exact required properties and schema to avoid these validation errors?

AWS Glue BigQuery connection (Go SDK v2) requires ConnectionType = BIGQUERY and a ConnectionProperties entry named “SparkProperties” whose value is a JSON string containing the lowercase key “secretId” that points to your Secrets Manager secret (which holds the GCP service-account JSON). Omitting SparkProperties or using a top-level “SECRET_ID” (uppercase) triggers the InvalidInputException you saw; also make sure your Go SDK/service model includes the BIGQUERY enum. Update the SDK if BIGQUERY isn’t recognized and store the GCP key JSON in Secrets Manager before creating the connection.

Contents

Quick summary of the fix (AWS Glue BigQuery connection, Go SDK v2)

Short version: set ConnectionType to BIGQUERY and put a ConnectionProperties key named “SparkProperties” whose value is a JSON string like {“secretId”:“my-bq-secret”}. The inner JSON must use the lowercase key secretId (not SECRET_ID) and that secret must contain your Google service-account JSON in AWS Secrets Manager. This structure satisfies Glue’s schema validation for native BigQuery connections and avoids the InvalidInputException. See the AWS connection-properties guidance and a practical example in the Pulumi docs for the same pattern: https://docs.aws.amazon.com/glue/latest/dg/connection-properties.html and https://www.pulumi.com/registry/packages/aws/api-docs/glue/connection/.

Why the “SparkProperties is missing” and “secretId is not defined” errors happen

Glue’s CreateConnection API validates a ConnectionInput against a schema. For native BigQuery connections Glue expects:

  • ConnectionType = BIGQUERY (enum)
  • ConnectionProperties to include specific keys used by that connector. For BigQuery the API expects a “SparkProperties” entry whose value is a JSON string. That JSON must include a “secretId” property pointing at the Secrets Manager secret with the GCP credentials.

Two common mistakes trigger the messages you saw:

  • Putting SECRET_ID (uppercase) as a top-level key in ConnectionProperties. The schema doesn’t accept that key and reports “secretId: is not defined in the schema”.
  • Leaving SparkProperties out entirely. The validator then complains “SparkProperties is missing but it is required”.

There has been historical confusion (and older SDK models) about which key is correct; if your SDK/service model predates Glue support for BIGQUERY you may also get an error that BIGQUERY isn’t a supported enum—update the SDK in that case (see the GitHub issue context: https://github.com/aws/aws-sdk-go/issues/5247). For runtime errors about the secret contents (missing service-account JSON) see the AWS re:Post example: https://repost.aws/questions/QUImFdxoyhRrqz9R3H5nCuZQ/aws-glue-bigquery-connection-launch-error-atleast-spark-hadoop-google-cloud-auth-service-account-json-keyfile-or-credentials-must-be-present-in-secretid-please-refer-logs-for-details.

Exact ConnectionInput schema — minimal JSON and Go shapes

Minimal JSON shape accepted by the Glue CreateConnection API for a native BigQuery connection (CLI/API form):

json
{
 "Name": "test-bq-native-conn",
 "ConnectionType": "BIGQUERY",
 "ConnectionProperties": {
 "SparkProperties": "{\"secretId\":\"my-bq-secret\"}"
 },
 "Description": "Optional description"
}

Notes:

  • The value of “SparkProperties” is a JSON string (escape quotes if you embed it in another JSON blob or shell).
  • The inner JSON must contain the lowercase “secretId” with the Secrets Manager secret name (or ARN accepted by Glue in many cases).
  • The Secrets Manager secret value should be the GCP service-account JSON (the entire key file content).

Equivalent Go SDK v2 structure (what you should use in your code):

go
ConnectionInput: &gluetypes.ConnectionInput{
 Name: aws.String("test-bq-native-conn"),
 Description: aws.String("Test native BigQuery connection"),
 ConnectionType: gluetypes.ConnectionTypeBigquery,
 ConnectionProperties: map[string]string{
 "SparkProperties": `{"secretId":"my-bq-secret"}`,
 },
},

If you used:

  • "SECRET_ID": "my-bq-secret" → validation error (secretId not defined)
  • ConnectionProperties: map[string]string{} with no SparkProperties → “SparkProperties is missing but it is required”

Working Go SDK v2 example (copy-and-run)

Replace secretName and AWS config/profile as needed. This example shows the minimal CreateConnection call that succeeds with the required schema.

go
package main

import (
 "context"
 "fmt"
 "os"

 "github.com/aws/aws-sdk-go-v2/aws"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/service/glue"
 gluetypes "github.com/aws/aws-sdk-go-v2/service/glue/types"
)

func main() {
 ctx := context.Background()
 cfg, err := config.LoadDefaultConfig(ctx)
 if err != nil {
 fmt.Fprintf(os.Stderr, "failed to load AWS config: %v\n", err)
 os.Exit(1)
 }

 client := glue.NewFromConfig(cfg)

 secretName := "my-bq-secret" // the Secrets Manager secret name containing GCP service-account JSON

 input := &glue.CreateConnectionInput{
 ConnectionInput: &gluetypes.ConnectionInput{
 Name: aws.String("test-bq-native-conn"),
 Description: aws.String("Test native BigQuery connection"),
 ConnectionType: gluetypes.ConnectionTypeBigquery,
 ConnectionProperties: map[string]string{
 // value is a JSON string that includes the lowercase "secretId"
 "SparkProperties": fmt.Sprintf(`{"secretId":"%s"}`, secretName),
 },
 },
 }

 _, err = client.CreateConnection(ctx, input)
 if err != nil {
 fmt.Printf("CreateConnection error: %v\n", err)
 os.Exit(1)
 }
 fmt.Println("Connection created")
}

Quick AWS CLI example (note quoting):

bash
aws glue create-connection \
 --connection-input '{
 "Name":"test-bq-native-conn",
 "ConnectionType":"BIGQUERY",
 "ConnectionProperties":{
 "SparkProperties":"{\"secretId\":\"my-bq-secret\"}"
 }
 }'

Common pitfalls & troubleshooting checklist

  • Wrong key name or casing:
  • Don’t use “SECRET_ID” at the top level. Use “SparkProperties” (top level) and “secretId” (inside the JSON string).
  • Value type:
  • The SparkProperties value must be a JSON string, not a nested map in the Go map literal.
  • Secret content:
  • Store the full Google service-account JSON in Secrets Manager (plaintext secret). If the secret doesn’t contain the key JSON, the job will later fail with runtime auth errors. See AWS doc for creating the secret: https://docs.aws.amazon.com/glue/latest/dg/creating-bigquery-connection.html.
  • SDK / model support:
  • If your Go SDK/service model does not recognize BIGQUERY (you get an enum error), update the aws-sdk-go-v2 packages to a recent release; older service models predate BigQuery support (background at https://github.com/aws/aws-sdk-go/issues/5247).
  • Permissions:
  • Ensure the Glue service role has permission to read the Secrets Manager secret (secretsmanager:GetSecretValue).
  • Debugging tips:
  • Try creating the connection in the console once to see what the console sends (then mirror that payload programmatically).
  • Use the AWS CLI JSON (above) to test the minimal payload.
  • If errors persist, copy the exact API error payload — it usually points at the offending property.

If you still see “secretId: is not defined in the schema” after applying the fix, double-check that:

  • You are sending “SparkProperties” exactly spelled/cased,
  • The value of SparkProperties is valid JSON string (no stray backslashes or malformed quotes),
  • You’re pointing at the correct secret name (or try the secret ARN).

Sources

Conclusion

To fix the InvalidInputException when creating an AWS Glue BigQuery connection with the Go SDK v2, set ConnectionType to BIGQUERY and include a ConnectionProperties entry “SparkProperties” whose value is a JSON string containing the lowercase “secretId” that points to your Secrets Manager secret. Update the SDK if BIGQUERY isn’t recognized and ensure the secret stores the GCP service-account JSON and is readable by Glue.

Authors
NeuroAnswers
Author
Verified by moderation
NeuroAnswers
Moderation
AWS Glue BigQuery Connection Go SDK v2: Fix SparkProperties & secretId