Intelligent Design : Architecture

Fellowship One REST API - OAuth Security Update

FTProductDev — Thu, 23 Apr 2009 07:43:00 GMT

NOTE: THIS UPDATE ONLY CONCERNS THE FELLOWSHIP ONE RESTFUL API BETA OAUTH IMPLEMENTATION

Yesterday a minor security threat was discovered in the OAuth protocol.

The OAuth community is aggressively working on the issue and have released an advisory update on this on 4.23.2009 at 12:00 am PST. We have been in contact with them on the details of this issue. After a close look at the threat and considering the risks involved, we have determined that we will not suspend the use of OAuth in our API. Instead, we will immediately implement additional mechanisms to further protect against potential attack:

Shorten the request token life time
Require any 3rd Party application to have a registered callback
Require additional warnings and details on the token authorization pages to clearly explain the intent of the Consumer application

We take all security-related issues seriously. Although the REST API OAuth implementation is only available in a beta environment right now, we want to take mitigating actions.These changes to our OAuth Service Provider implementation will be deployed soon.

We purposely chose OAuth as our primary authentication mechanism because of the security and iron-clad structure it provides. We are even more encouraged by our choice in light of the quick response and reaction from the OAuth community. Since there are thousands of developers and applications behind this protocol, there are thousands of people behind our implementation.

We would like to thank Chris Messina, Eran Hammer-Lahav, and the rest of the OAuth community for being so transparent and quick to respond and collaborate to solve this issue.

Please do not hesitate to contact us directly with any concerns or questions that you may have: api [at] fellowshiptech.com

More information: see OAuth advisories updates

Nick Floyd
Integration Architect

Fellowship One APIs - Past, Present, and Future

FTProductDev — Wed, 25 Feb 2009 22:42:00 GMT

With the up coming Fellowship One RESTful API we wanted to take the some time and look back on our other two APIs and reflect on what we intended them to be and what they actually became. We looked at the their interfaces, authentication, resources, versioning, protocols, logging mechanisms, "debuggability", etc... What we found out was that in the world of "API" nothing ever ends up like you intend.

Fellowship One Data Exchange: A very powerful XML RPC API that was specifically designed to address massive data import and export needs. It's ridged pinpoint interface allow users to build simple data presses, which move large amounts of data between disparate systems. A few years into the product's life we introduced the ability to do synchronous transactions - a decision that completely changed many consumer's usage patterns.

Serendipity stepped in and instantly consumers began to use this brute force application on websites to perform anything from profile updates to seamless login patterns. After seeing how much the community found value in the login metaphor we've published docs and given presentations on the best practices of this approach. We now have 3rd parties writing web applications and hosting web sites that do a myriad of things.

Being a SOAP based .NET web service, consuming Data Exchange required the language to either have a library that constructs SOAP envelopes or that the developer understand that protocol and be able to construct a SOAP message manually.

Another characteristic of this API are the logging and response mechanisms; both were intended to only be used / interpreted by the consuming application. Its authentication is based on simple credential / key maps.

Fellowship One Payment Gateway: A .NET SOAP based web service that gives churches the ability to submit giving / e-commerce transactions. The interesting thing about this service is that it was originally developed for internal consumption only and later opened up to external consumers for book stores, tithe applications, etc... another decision point that changed the way an established API was used.

Payment Gateway answered a changeling question of: "how do I get all of my billing and payments to go through one channel and be able to report on the transactions going through that channel?" Much like Data Exchange, we saw it consumed in ways we couldn't have imagined.

We used credential and signed key authentication for this API.

Fellowship One RESTful API: A REST based web application that uses several open protocols and patterns to provide consumers access to secure resources.

STANDARDS, PROTOCOLS, and PATTERNS - nothing else. Looking back, we could see where their is major value in sticking with something that is "tried and true." If we could use web based patterns and protocols for a web based API we could not only get instant adoption but gain the efficiencies and effectiveness of technologies that "just work."

This next generation of API is, ironically based on foundations that have been in place for years. We chose MVC as the application architecture, OAUTH protocol for Authentication, REST and HTTP 1.1 as the transport. This API should give web developers an tool where all they have to think about is what to do with the resources.

Thank you for partnering with us over the years; we have a lot of fun stuff up ahead and we hope to see you at the Dev track @ DC09!

Innovating with MVC in ASP.NET

FTProductDev — Thu, 01 Nov 2007 00:55:00 GMT

If you were following our blog at the end of last year, you read my entry on MVC - Wrestling an Enterprise Architecture out of .NET. Well, we pressed forward and developed a fully-working MVC2 architecture on top of ASP.NET. We've had some bumps along the way, but we think it's been worth the effort to move to an MVC2 architecture. The benefit of moving the majority of your application's logic into controllers cannot be overstated. Testing and API extensibility are just a couple of the huge benefits. We said as much to Scott Guthrie in an email conversation at the beginning of this year, and asked why Microsoft didn't address this architectural need.

We were very pleased to hear that Microsoft is going to do just that with their MVC framework in ASP.NET, and it feels good to know we were ahead of the curve. Matt Vasquez and I also had the honor of attending a Software Design Review of MVC and some other technologies, along with about 20 others, in Redmond last week. It was a great event, with tons of info on upcoming technologies and some good conversations with the developers. Not to mention the fact that a thirty second question to Scott Guthrie after dinner solved a problem that our development team has spent at least a collective 24 hours trying to figure out. While everything covered in the SDR is under a non-disclosure, suffice it to say that we're excited about this new MVC support from Microsoft. Our home-grown architecture is very legitimized by what we saw. We're very close to what they've developed, but I'm sure we'll be able to use a lot of what they're creating to more cleanly handle what we had to hack onto ASP.NET. It's also exciting to be able to influence the development of this new framework. I plan on cranking some code from the SDR bits starting this weekend and giving lots of juicy feedback.

I pray that all our customers know that we place a great deal of importance on innovation. It's a high priority for us to innovate both our architecture and our processes to improve our product, and we aren't afraid to go to the lengths necessary to align our development with future-thinking choices. It takes a pioneering spirit to not only see where technology needs to go, but to also take the effort/time/risk to move in that direction. This past year, we moved to the Agile process of Scrum and created our own MVC architecture on top of ASP.NET. All of our new development now adhears to the MVC pattern (MVC2 if you want to be specific). God has given us a pioneering spirit, and we're excited about where He's leading us.

thardy

Typed Navigation

thardy — Tue, 09 Jan 2007 17:12:00 GMT

One of the features we've worked into our new home-grown architecture is typed navigation. Without typed navigation, urls are strewn throughout the code, and any changes to navigation or querystrings involve updates to many pages. Not only that, providing a page the data it needs to properly initialize can be difficult to manage and very easy to misunderstand by a developer who is working in a page they're not familiar with. These issues cause navigation edits to be error-prone and reduce maintainability and extensibility.

The Need

Consider the following common scenario - You've got a page (let's say EmployeeDetail.aspx) that requires an EmployeeID be passed to it. There are many different ways to get the ID to the page:

QueryString - EmployeeDetail.aspx?empID=100
Session - Session["EmployeeID"] = 100
Database retrieval
etc...

Each method involves different code to set and retrieve the value. In addition, almost every app uses multiple methods in different places and in different scenarios. This results in too many decisions being made by too many different developers, which results in maintenance nightmares. Strict processes can mitigate some of the maintenance pain, if all of your developers ALWAYS use the exact same mechanism, but what about extensibility? What if the page name needs to change at some point? What if you decide to change the way you handle data transfer between pages? Or, more commonly, what if a page simply changes the data it needs or accepts?

Cascading Style Sheets (CSS) is a good example of an architecture with big benefits in UI design - you can make a style change in one place, and that change gets applied in all pages where that style is used. The same measuring stick can be applied to all architectures - can you make a configuration change in a few places (preferably one), and have it applied in all areas of usage? I think most apps would fail this measure in the area of navigation. If a page name changes, most application architectures would require manual changes in many pages - links, code behind redirects, data tables, etc. Even more architectures would fail the test if a change is made to how data is transferred. If a QueryString value changes from empID, to eID, how many code changes would need to be made? What if a page now requires an orgID? Or how about if you move from QueryString to Session, or vice versa? This is where good architectures shine in the long run. Yes you can continue to change the font manually in all your pages when your app is small, but heavy usage of CSS is now widely accepted as best-practice simply because style maintenance and extensibility become absolutely crazy as the number of pages in your site gets large. Why should navigation be any different? It gets just as crazy as your app gets large, and can benefit just as much from a consolidated architecture.

The Solution

Our solution is to use typed navigation. There are several potential architectures available, one of which is PageMethods. There's a great list of the benefits of typed navigation on that page, so I'll try not to rehash all of them here. We took a good look at this solution before developing our own architecture, but decided we wanted much more control over our navigation. Also, the PageMethods solution is locked into generating urls with QueryStrings, and it involves pages calling specific methods in other pages, which our new architecture steers clear of due to MVC principles.

Our architecture includes the following:

Typed Location objects - every navigation destination (View) has its own Location class
Each Location class defines a constant "ViewName" which maps to a url defined in a database, so no urls or page names are ever written in code
Each Location class can have multiple constructors - the constructors provide all the contracts to create the View and allow pages to fulfill multiple roles (list/detail based on what data is sent to them)
Usage in links - a new Location object is created in code-behind and used to set the Location property on one of our HyperLink controls. If set, the HyperLink control has the Location generate its URL when the control is rendered.
Usage in server-side redirects - a controller simply creates a new Location object and calls a public Navigate method on its View
Hydrating Location objects - our base Page class takes care of rehydrating the location object and transferring its data to the View

Here are the only two options a developer will see regarding Navigation in our app:

For a HyperLink:

lnkEmployeeDetail.Location = new EmployeeDetailLocation(employeeID);

For a server-side redirect:

_myView.Navigate(new EmployeeDetailLocation(employeeID));

Since all this is handled by the architecture, changes can be made in just a few places and totally change the Navigation mechanism. We can switch from QueryStrings to Session variables, or rewrite URLs, or change how we rewrite URLs as often as we want. No code that actually uses the Navigation would ever need to change, and page names can change without missing a beat. All the code written to create strongly-typed Location objects would continue to navigate as usual. If a page's initialization requirements changed, then code would break at compile time instead of runtime, with a very clear and self-describing explanation of exactly how the new Location object needs to be constructed. That's a lot better than having some forgotten link deep in your app with an obsolete parameter definition getting discovered by a customer after you've published your app to production.

Typed Navigation is one more step we're taking to make our code easy to work with and understand, regardless of how long it's been since anyone's touched it, how many developers have had their hands in it, or how large it grows.

Persistence via Attributes

thardy — Fri, 20 Oct 2006 13:12:00 GMT

In the new architecture we've been developing, one of the features we've developed is a common persistence mechanism. Once again, our attempts are to make development as clear and painless as possible. We want to make it incredibly easy to write good code, and difficult to make mistakes. As our development team grows and code authors move on to other things and other devs end up maintaining that code, we want it to be downright difficult to misunderstand or misuse anything.

Persistence is one area that we've seen misused in the past. Basically, anyone can put anything in Session or ViewState from almost anywhere they want. Well, they can from the code-behind of pages, which, like I discussed earlier, ends up getting a great deal of logic thrown in there. We decided to put a stop to this in our new architecture.

We've refactored all the persistence code into base classes, and we're using attributes to decorate our View properties like the following:

[Persist(PersistenceType.Session)]
public int IndividualID {get{} set{}}

This allows us to very simply specify what needs to be persisted and how. Currently the three different methods of persistence we're starting with are - ViewState, Session, and Process.

ViewState - this one is pretty self-explanatory

[Persist(PersistenceType.ViewState)]

Session - this is for page-level session only. It persists the data for as long as the user doesn't navigate to a different page. The fact that we've refactored all our persistence code allows us to implement a very aggressive state management strategy that deletes any session data that does not EXPLICITLY belong at any given time.

[Persist(PersistenceType.Session)]

Process - this is for state that needs to be maintained across pages. If a page wants to enlist in a shared-state scenario (like a wizard or any sequence of pages the user can move back and forth between). The developer just passes the name of the process to the attribute's constructor, which assumes a Process persistence. Once again, due to aggressive state management, we immediately delete any process data that isn't explicitly enlisted in by the currently loading page.

[Persist(ProcessList.RegistrationProcess)]

For state that needs to be persisted across the user's entire session, our base Page class will maintain this responsibility. No other code should have this capability. Also note that nothing in our Controllers or business layer can push anything to persistent state. Those layers deal with explicit instructions and methods, some of which may store things in the database, etc, but there's no concept of a Controller saying, "I want this to be placed in persistent state". Controllers just manipulate properties and expect them to be stateful. It's up to the presentation layer to decide how something should be persisted. The difference between WinForms and WebForms is so great that the idea of placing that logic in the Controller makes no sense to us.

As to the performance issues of using Reflection to read and act upon attributes, we feel it's negligable compared to the benefits. We ran some simple tests, and found that using Attribute-based persistence was approximately 100 times slower than not using Attributes. While that may sound huge, it's not when you're talking about the difference (in the worst case) between .00281 and .000228 seconds. Or, in a longer running scenario, between .000268 and .0000363. Not only that, but we've researched several methods to make long-running reflection usage far more performant via the use of Reflection.Emit and also ASP.NET 2.0's Dynamic Methods. Once we've solidified other areas of our new architecture, we can go back and tighten the screws.

These changes will increase the visibility and maintainability of our code by allowing any developer to very quickly see exactly what is persisted and how. Having all our persistence logic refactored into one place will eliminate any potential mistakes a developer could make in persisting data, and will increase our agility in extending the functionality in the future. The point is, there's too much to gain from Reflection not to use it.

Wrestling an Enterprise Architecture out of .NET

thardy — Fri, 22 Sep 2006 23:24:00 GMT

As a growing "Software as a Service" (SaaS) development shop using .NET we've encountered many growing pains and some of the speed bumps inherent in the architecture that many others probably have as well. I'd like to express some of the issues we've encountered and hear if anyone else would like to start a dialog on the shortcomings and potential solutions to scaling enterprise .NET development.

The bigger the code, the bigger the problems

The typical architectures espoused by .NET aren't well-designed for large, "always-on" apps. This has become clear to us as our development group and our codebase have grown. SaaS is a big factor here as well, since best-practice is quickly discovered to be a rather tight, incremental release schedule. As our development group scales, and feature demands increase, the framework everything is written in begins to show its strengths and weaknesses. It's the sort of thing that is difficult to see until many hands have been in the same pot over a period of years. J2EE seems much more geared towards large, enterprise apps, with more emphasis on strict contracts and black boxes. While many of the Java frameworks seem like overkill to most .NET developers, it's because most .NET developers don't work on the same codebase for more than a year or two. It seems like Microsoft has made a calculated move to aim squarely between the enterprise and the PHP-type developer communities. We love .NET tools though, and are looking at some foundational architecture changes to tackle these issues.

Misplaced intelligence

One of the key difficulties introduced with .NET is the Page-Controller Pattern. Despite how nice and elegant the most common n-Tier .NET architecture sounds on paper, practice shows that a great deal of intelligence (WAY TOO MUCH intelligence) inherently ends up in the code-behind of your pages and user controls, and thus, smack in the middle of your presentation layer. This produces a tight coupling between the presentation and business layers and a lot of business logic in the wrong place. The commonly espoused reason for decreasing this coupling is the ability to completely change the UI layer - say, to take your existing business layer that you used with your Web pages and slap WinForms directly on top of it. While this would be wonderful, the more pressing reason is the simple need to refactor code for maintainability and extensibility. Anything in your code behind pages cannot be reused elsewhere. While having nice business methods in your business layer like "FetchOrdersByCustomer", or "UpdateProducts" that deal with nice typed objects and collections may make you feel all warm and fuzzy because you have your n-tier bases covered, your architecture bases really aren't covered at all. Or maybe you have your entities possess the intelligence to manipulate themselves. Either way, the majority of custom development occurs within the consumers of these methods. If the consumers of these methods are in your code behind, you will have real problems as your product grows, your development team grows, and the demands on your product begin to pull it in multiple directions. What you really want is an incredibly stupid presentation layer that only has intelligence about itself. It should only know how to present information from a strictly contracted model it knows nothing about manipulating, how to manage state between "locations" (pages, winforms, pocketPC forms, etc), and how to handle client-side behavior. This is another argument for stupid entities as well (our entities do not know how to manipulate themselves), since our presentation layer can reference our entities without having to worry about exposing all that business logic to the presentation layer. I'm sure I'll get some disagreements on that, but that's the beauty of hearing other people's opinions.

A possible solution

The solution we are developing is moving towards something closer to the Model View Controller pattern. Particularly MVC2, which is the same pattern altered for the web. Microsoft's User Interface Application Block is an MVC block, but MVC does not work on the web. MVC depends on the View being directly updatable from changes to the model, so that combined with the complexity of that application block make it unfeasable for our use. Since the web is stateless, there is no way to directly update your UI from changes to the model. What MVC2 does is simply insert the controller into the process coming from both directions (web to model, and model to web). Microsoft has no packaged solution for anything close to MVC2, which is really disappointing, since you would think they'd want to compete more directly with J2EE and fill this gaping architectural hole. There are some packaged solutions from third parties (even more evidence of a serious development need unmet by Microsoft), such as Maverick.NET http://mavnet.sourceforge.net/, MonoRail http://www.castleproject.org/index.php/MonoRail, and something called NWAF http://sourceforge.net/projects/nwaf, but all of the above take rather radical departures from the .NET way of doing things and are a big jump for .NET developers. Things like the basic page lifecycle and ViewState are radically changed. We wanted something that wasn't quite a big leap for our group, yet still gave the major benefits of MVC2.

There has been a pattern that has recently risen to the top of the .NET pattern circles called Model View Presenter (MVP) that attempts to take a stab at MVC2 and still maintain WebForms and ViewState management. I've recently read a few articles that mention Microsoft's CAB (Component UI Application Block) will have some MVP patterns in it, but we're not holding our breath. It's also targeted at Smart Clients, so I don't know how practical it is for pure web-based apps.

We're building our own home-grown version of an MVP architecture for all development going forward. We are also baking in a few more things that we feel are important, such as:

Typed navigation - urls should never appear in code, and it would be nice if the developer knew the exact state data required in order to navigate to that location/view at compile time
Removing all state-management concerns from the Controller layer - controllers should not care anything about how state is persisted or even have to make the decision that something should be persisted. They should just be able to expect the properties on the View to be available.

As we get further along, I'll post some of the pitfalls and breakthroughs we come across.