lambda-arch | Applying Lambda Architecture with Spark Kafka

One important thing to keep in mind is the 2 sinks will be used from 2 distinct queries, each reading independently from the source. So checkpointing is done per-query.

Whenever you call start on a DataStreamWriter that results in a new query and if you set checkpointLocation each query will have its own checkpointing to track offsets from the sink.

That should be possible using the -T option of pg_dump:

-T table
--exclude-table=table
   Do not dump any tables matching the table pattern.

The psql documentation has details about these patterns:

Within a pattern, * matches any sequence of characters (including no characters) and ? matches any single character. (This notation is comparable to Unix shell file name patterns.) For example, \dt int* displays tables whose names begin with int. But within double quotes, * and ? lose these special meanings and are just matched literally.

A pattern that contains a dot (.) is interpreted as a schema name pattern followed by an object name pattern. For example, \dt foo*.*bar* displays all tables whose table name includes bar that are in schemas whose schema name starts with foo. When no dot appears, then the pattern matches only objects that are visible in the current schema search path. Again, a dot within double quotes loses its special meaning and is matched literally.

Advanced users can use regular-expression notations such as character classes, for example [0-9] to match any digit. All regular expression special characters work as specified in Section 9.7.3, except for . which is taken as a separator as mentioned above, * which is translated to the regular-expression notation .*, ? which is translated to ., and $ which is matched literally. You can emulate these pattern characters at need by writing ? for ., (R+|) for R*, or (R|) for R?. $ is not needed as a regular-expression character since the pattern must match the whole name, unlike the usual interpretation of regular expressions (in other words, $ is automatically appended to your pattern). Write * at the beginning and/or end if you don't wish the pattern to be anchored. Note that within double quotes, all regular expression special characters lose their special meanings and are matched literally.

Acutally when you execute the vagrant up in the console, it will show the download processes.

But for your question, all the downloaded boxes are house in "C:\Users\USERNAME\.vagrant.d\boxes" folder.

Baically due to the poor connection, vagrant download the boxes very slow, so it is high recommand to download your base box in http://www.vagrantbox.es/ or https://app.vagrantup.com/boxes/search with the download tool, then you can add it by

vagrant box add vagrant init vagrant up </code></p>