• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!


Beyond Traditional Architecture

Page history last edited by Robert Ream 15 years ago

I have re-orded some of the content to keep it organized. I'm not done either, but feel free to update.


What's Oren Thinking About


Architecture for scale: 1k -> 10k -> 100k users without having to rewrite

  • Ex.: Customer won't use NHibernate because they can't change the SQL it generates. Database server is using different query plans for different sized datasets.
  • Decouple processes so they don't all have to respond at the same time.
  • Map Reduce in .NET: Dare Obasajo's post.
  • Reduce complexity - complex SQL, "select is broken"
    • ex.: customer has complex tree with ~ 100k items, 40s complex query to fetch items, changed to using application logic to filter and fetching when subtree opened, reduce to 5ms.



  • Constraining scale: i.e., not having a limit clause on a query is equivalent to not disposing a disposable resource, which is a bug
  • Google app engine has constraints on how long your application can take to complete a request.
  • Active constraints, such as # of queries per request



  • A database of name value pairs.
  • You can create views on top of this data: To deal with data access patterns like "...over 10 years" vs. "...over one day" you would create different views.
  • Forces you to think about queries / data access patterns instead of data model


Repository as a Leaky Abstraction

  • Unbounded result sets are the problem.

Alternatives to RDBMS-Classic

  • Transaction Log (data access patterns: CR, no UD):
    • Write to files.
    • Offline/async process extracts data from files for other processes, like reporting, etc.



Follow on session on Sunday:

For some reason the java guy mentioned that there are Java things the .NET world should look at and know about.


- Apache Gump (continuous integration at new levels)

- Hadoop (Map/Reduce in Java)

- Maven and Ivy (project meta information allows for inter-project build, deploy, etc.)

- Hudson is an awesome CI tool that comes from java world, but would work for .NET projects also


Why aren't there similarly-sized p[rojects in .NET?

having something like ruby gems repository or java ibiblio repository of projects, artifacts, would be good for .net


Asked the room who was doing applications where this kind of scaling would be needed - Bryan Hunter talked about his citizen journalism application - lots of raw media coming into the system, many local TV station affiliates pulling media from the system. Scales by having nodes in parallel - can move an affiliate to its own node. Using NHibernate


Oliver Champet discussed Drillinginfo applications - using Hibernate to read data in - large selects, iterate over result set and process each row. Hibernate to merge data back into the database, but right now blocked by legacy DAOs with business logic, triggers, sprocs.

Jeremy Miller suggested that as DrillingInfo moves this way, look at writing domain specific query language - maps to HQL. Find and extract business logic from DAOs and database into code. Use characterization tests to lock down existing logic, then keep the tests green as logic gets refactored into code.

Jeremy mentioned that there is a project afoot to do LINQ-like features in Java - Quaere

Steve Donie mentioned that they were using dbFit (FitNesse plugin) to do database characterization tests.


Discussion around alternatives to RDBMS

- other repositories for site specific configurations

- are there any OO databases for .NET?

- not specifically .NEt - http://www.intersystems.com/cache/index.html no one here has any experience with it

- lighthttpd

- use onion-type arch so that your app isn't tightly coupled to things like (for example) LINQ to SQL 

- DB4o (Greg Young) "Simply embed db4o's open source object database engine into your application and store and retrieve even the most complex object structures with only one line of code."

- Sharon mentioned a project where rather than using RDBMS for doing large scale searching, export to flat files and use grep (for smaller stuff) or Lucene (for larger stuff)

- Amazon S3, Limelight networks and similar CDN (Content Delivery Networks). Akamai, etc.


Jeremy mentioned QCon - folks should plan on going.


- Domain Driven Design with C#

- learn SOLID principles.


Starting to see a similar impedance mismatch as we've seen between Objects-Databases with the Object-Functional space. Perhaps there is room for doing things where we put the functional stuff behind an interface - use it as a service for the things where it is appropriate.


Using GPUs as parallel processors - http://www.manifold.net/video/Supercomputer_GIS.wmv



Action Items:

Read through this. Absorb all. Report back to your cow orkers.




Comments (0)

You don't have permission to comment on this page.