Skip to content

how to read flattened type field from elasticsearch using pyspark connector? #41

Description

@amendhe

After reading index from elasticsearch using pyspark in databricks - All fields appeared but only one field which is a type of 'flattened'. Is there any option that i need to include during read. Following is my snippet to read from index.

spark.read.format("org.elasticsearch.spark.sql")\
      .option("es.nodes", ",".join(db['nodes']))\
      .option("es.mapping.date.rich", "false")\
      .option("es.net.http.auth.user", 'abc')\
      .option("es.net.http.auth.pass", '123abc')\
      .load('indexname')

elasticsearch: 7.5.1
spark: 2.4.3

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions