[talk] AWS - In memory databases

[Edward Capriolo]

Hive uses a meta store (normally mysql)  to store the schema information
including location of data, schema of data.
If you only have a small about of meta data and you can recreate it quickly
people will do it in the startup process of there emr process.
If you have thousands of tables with thousands of partitions uses a
reliable database you know how to backup and manage.


On Mon, Oct 21, 2019, 12:55 AM Sujit K M <kmsujit at gmail.com> wrote:

>
>
> On Mon, Oct 21, 2019, 1:31 AM Edward Capriolo <edlinuxguru at gmail.com>
> wrote:
>
>>
>>
>> On Sun, Oct 20, 2019 at 11:23 AM Sujit K M <kmsujit at gmail.com> wrote:
>>
>>> Always been a critique of AWS. Found something that might fuel that more.
>>> Not sure whether there will be more acceptance.
>>>
>>> Below an article on something I was researching.
>>>
>>> http://www.devgrok.com/2018/12/using-s3-hive-metastore-with-emr.html
>>>
>>> Below an article on In memory Database on AWS
>>>
>>> https://aws.amazon.com/nosql/in-memory/
>>>
>>> As per some of the discussions we have had on EC2 on FreeBSD. I find
>>> supporting something like HSQL(Java In Memory Database) on AWS very
>>> difficult which is certified by other cloud vendors, they seem to think
>>> how
>>> can you not have it.
>>> _______________________________________________
>>> talk mailing list
>>> talk at lists.nycbug.org
>>> http://lists.nycbug.org:8080/mailman/listinfo/talk
>>>
>>
>>
>> You are looking at vastly different technologies.
>>
>> Hive is batch processing system with an SQL frontend. At best it works at
>> analytic speed. Queries over GB of data in the seconds to minute range.
>>
>> This is different then single node "in memory" databases. (HSQL)
>>
>> Which are different then multi-node distributed in memory databases
>> (VoltDB)
>>
>> Aerospike and redis are KV like stores without relational capabilities.
>>
>
> Not sure whether you have read the link. It states having MySQL as backend
> to store hive data. My question is what ever is in s3 bucket, do they run
> out of box.
>
>>
>> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org:8080/mailman/listinfo/talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20191021/8b0adb7f/attachment.htm>