Posts

Demo Day!

Doing a demonstration of OPUS in Florida.

This site (openplatform.us) is known as a node. It is an independent site and it is independently owned and operated. It is also independently funded.

I've made it very easy for people to operate their own nodes. I've also made it very easy for people to connect their nodes to one another and share information.

The world's first fully decentralized real-time information network is almost here. All I can tell you at this time is that you're not going to recognize this site in a few weeks. My hope is that you won't recognize the Internet shortly thereafter!

-Rob

Imagine opening the biggest can of whoop-ass, setting it down and walking away.

About Assumptions

One of the things people like to do is assume. They simply "press play" on what they believe to be true and start typing. And, usually, their intuition or instincts were correct. They get away with it...usually.

Then, once in a while, people step in some shit they just don't understand. It has an odd smell. It doesn't wipe off the bottom of their shoe.

What you're looking at right now as openplatform.us is the prototype. This site was a proof-of-concept meant to do a better job of explaining a concept than a static document. The project has since become much more and is headed in new directions.

I don't ask people to believe anything. I keep typing and get shit done. You can keep assuming. That's actually the fun part of doing it my way. The assumptions provide endless entertainment.

I've had full document structure with multiple embeds since 2014. I blew 140, 280, 300 and even Facebook's limits out of the water with 16MB per post. 16 million characters and your choice of delectable emoji should cover the communication needs of most. Basically, everyone gets a blog. A massively distributed blog.

Or, you know - keep doin' that bitch shit you do. I do not care.

Optimizing Analytics

ORACLE has materialized views, a technique for pre-generating reports for write-once/read-many use cases. They maintain themselves, which is a convenience and optimization in and of itself.

For those who aren't fortunate enough to be working with ORACLE as their database and for those who find materialized views to be limiting, MongoDB offers the Aggregation Framework. It's as if the simplicity and joy of a document-oriented solution wasn't enough. They're like: Here! Have some advanced reporting, too!

MongoDB Aggregation Framework

The MongoDB Aggregation Framework, as its name might suggest, allows a developer to "aggregate" the data in a collection into a report. At my new company, I use the Aggregation Framework to do things like count the number of tweets that appear in the news per screen name (Twitter user).

If you are wondering where SQL's GROUP BY statement lives in MongoDB, it's called the, "Aggregation Framework." Seriously - you get a whole devkit centered around the concept of grouping data. It's nuts.

Aggregation Pipeline

In MongoDB, aggregations are performed entirely within the database in stages in an object known as the Aggregation Pipeline. The aggregation pipeline itself is simply an array.

Each element within an aggregation pipeline array is known as a stage. Each stage defines work to be performed on the data of the collection. The developer envisions data entering at the first element of the pipeline and having an operation or operations performed on the data. The result is then passed to the next element in the pipeline. This process continues until there are no more stage elements in the pipeline.

The Aggregation Framework receives one complete "pipeline" as its work order, and the entire pipeline is executed within the database. This is a radical optimization for document-oriented storage systems, which tend to reply on the application server to perform tasks such as aggregation. The Aggregation Framework is definitely one of the feature sets within MongoDB that set it apart from key=value systems such as Redis.

Advanced Options

MongoDB's Aggregation Framework now has some incredibly advanced features such as being able to use the disk for processing and also storing aggregation pipeline final results into a new collection in the database. That last bit, represented by the $out pipeline operator, is what has given MongoDB the equivalent of Materialized Views from the relational world of ORACLE.

Materialized Views vs. Aggregation Framework

I don't often like to discuss the performance differences between MongoDB and relational systems because they are two different tools doing two different jobs. Professionals don't do things like compare a dentist's drill to a sledgehammer. It's stupid. Stop it.

The main differences I'll discuss between relational systems and document-oriented systems are productivity boost for developers and additional developer concerns. When comparing Materialized Views to the Aggregation Framework, we have one major point to consider.

MATERIALIZED VIEWS ARE DEFINED IN THE DATABASE.
AGGREGATIONS ARE NOT DEFINED IN THE DATABASE.

What I mean is that the "Database Administrator" actually defines a materialized view within the database. ORACLE basically allows a developer to write a query that stores its result in a new table. This is known as a materialized view because the results are actually persisted to disk unlike a "view" - which is simply a query that is executed every time the view is queried. MongoDB, to the best of my knowledge, has no equivalent to a "view" because they are fucking stupid.

In a system such as ORACLE, a materialized view is a query that lives within the database and produces a "view" of that table. Additionally, as the source table for which the materialized view is defined is updated, ORACLE ensures that the materialized view is also updated. ORACLE does this atomically. When the write operation completes, the application developer knows both the source table and the materialized view are updated.

In the MongoDB Aggregation Framework, the developer does not define aggregations within the database. MongoDB accepts no responsibility for ensuring that an aggregated result set is updated. Ever. In fact and by default, MongoDB doesn't even keep the results of an aggregation. They are intended to be reports.

So, when aggregations first arrived, they were very limited. They could only contain up to 16MB of data as a single "document" within the MongoDB vernacular, and all processing was performed in memory. These were rather limiting features that kept aggregation from being widely accepted. Things have matured since.

Aggregations can now (optionally) make use of disk storage while doing their processing. They can also atomically write their result sets to new collections. And, while I wouldn't build a system responsible for keeping human beings alive with it? I can vouch that it's good enough for social media analytics. Because that's how I'm doing it.

At the application level at the conclusion of RSS feed updates every 10 minutes, I kick off an aggregation of my rssfeedentries collection to scour the news for tweets. This aggregation generates its report in a new collection on disk, which is then served out by the front-end application tier. There are some additional restrictions when writing your results to a new collection. The Aggregation Framework cannot write to a sharded collection. It also cannot write to a capped collection. Because God-dammit, fool, grow up.

Besides, if your reports are so damn massive that you'd need to shard them for performance, you can definitely find a way to simplify that report and offer drill-downs, etc. You do not have to put everything in one query, you can learn to use new tools and be a happier person. I'm a happier person. And, I'm a happier person because MongoDB's Aggregation Framework does not suck.

Learning More

Go here: https://docs.mongodb.com/manual/reference/operator/aggregation-pipeline/

Post Title

This is a quick post about nothing. I am doing this post as part of a demonstration while broadcasting live online. I am on Periscope right now. I don't want to be on Periscope right now.

This is a second paragraph.

Remember: No matter how banned I am on social media, I will always have a permanent home here. And, I'm looking at dramatically re-vamping this whole site into something a little bit better.

I want a strong departure from "social" networking. It sucks. It's always sucked. It has no purpose. This is an information network. And, no - I don't care if that's "strange" to you. I am not #metoo.

I swear: One of these days, I'm going to finish this. I will find the time.

The arrogance is stunning as #DWS pushes the Clinton single-payer/universal health-care-as-a-right agenda. Really can't wait to see all the pay-to-play, espionage, murder and pedophilia charges coming out of her laptop.

Debbie Wasserman Schultz was pretty much hand-picked for politics, placed into a leadership role at the DNC and instructed to ensure that Hillary Clinton was the 2016 nominee. It's her role. And, she had to break a ridiculous number of laws a ridiculous number of times to make that happen. But, she did that. She did as told. She played her role.

Her secondary role is to do everything possible to advance the Clinton agenda in our federal legislature. This is a video of her doing as told and playing that role to a tee. She even goes so far as to "reclaim her time" like a certain other amazingly well-documented corrupt Democrat: Maxine Waters.

Had a wonderful conversation today with some people involved in real-time transcription services. Mind = blown. Truly.