Think Different

  Photo by Gül Işık Handling large binary data in Azure Synapse When dealing with large binary data types like geography or image data in Azure Synapse, you may encounter challenges due to limitations in supported data types and column sizes. Let's take the example of a City table with a Location column holding geography data, which needs to be converted to a varbinary type during loading since Azure Synapse doesn't natively support geography types. Example: 1. Convert to varbinary: During loading, convert the geography data to varbinary. 2. Data Chunking: Since PolyBase supports varbinary up to 8000 bytes, data may get truncated. To overcome this, split the data into manageable chunks. 3. Temporary Staging: Create a temporary staging table for the Location column. 4. Chunk Processing: Split the location data into 8000-byte chunks for each city, resulting in 1 to N rows for each city. 5. Reassembly: Reassemble the chunks using T-SQL PIVOT operator to convert rows into colum...