LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dglavastorm

Hi, We've just started using Tableau as our front end and are now using the "output Tableau" TDE publish node. Is there any way to import a TDE ?

We'd like to create the TDE and then append when new data becomes available, which would require reading the TDE back in to LS then overwriting the TDE file with the updated data.

Thanks,
Dave

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

Hi Dave,

the Tableau Data Extract (TDE) file format is Tableau proprietary. Looking online, there have been a number of requests for Tableau to open source the TDE formats so that other applications can read extract files. However, there is a statement on the Tableau community website that says: "The existing TDE API will continue to create TDE extracts but we have no plan to open source it."

The post mentions that the new Tableau Hyper extract will replace the existing Tableau Data Engine and that "there will be an API for creating and updating Hyper extracts". No details of the API are available at this time.

Regards,
Adrian

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: gmullin

If you can't easily read the TDE file an easy solution would be to write/append the data to a BRD file at the same time you write the TDE file. Then you can easily read back in whatever you had in TDE by using that BRD file.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dglavastorm

Thanks all, curious how Lavastorm can create a TDE but not read one ?

Also the second issue underlying this is that the data sets I'm working with are massive. Using BRD or CSV the file is over 5GB, but LS compresses this down to 100mb as a TDE. I'm trying desperately to balance the required work with a workable storage solution, and I can't shrink the underlying data any further.

So I guess a secondary question is, if we can't read from a TDE, is there any other format I can use that heavily compresses the data ? XLSX won't work, the node just doesn't run even if I adjust the JvmMaxHeapSize, and I would prefer it in a single file.

Data Warehousing sadly is not an option.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Originally posted by: dglavastorm
					

Thanks all, curious how Lavastorm can create a TDE but not read one ?
			

We use a library of code provided by Tableau - that library can write a TDE, but can't read one.
In simple terms, Tableau themselves haven't provided a mechanism for reading, and their format is proprietary - they have not yet provided information sufficient for other companies to read the TDE files.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dglavastorm

Makes sense, thanks.

So are there any formats the LS can write to that offer good compression ? And I don't mean compress to an archive like ZIP, it needs to be readable.

BRD didn't really compress, it just made reading the data back in faster.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: gmullin

I think the answer to your question is no. I tried to write out a BRD file of around 50mb that as TDE was 50kb but any other format I tried the file was of similar size if not larger. The only other option is to compress the file, but you said that's not an option for you. Maybe somebody has a custom node out there? I'd be happy to be proven wrong.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dglavastorm

Thanks, that unfortunately matches everything I've tested as well, including using a Teradata datalab. Whatever Tableau is doing, their compression is sensational.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dstacey

Are there plans to upgrade the Output Tableau node to support the new hyper format? Thanks.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

Hi,
we have no plans to provide support for the Tableau .hyper extract format on the Lavastorm product.
However, this may be something that could be considered for the Dataverse product.