Developing solutions for SharePoint Online

By now, you should have a basic understanding of what SharePoint Online is, how information is structured within SPO, and how you can provision your own developer site that can be used as a playground for your upcoming custom deployments.

Next, we'll look at what kind of solutions developers can build for SPO. We'll start with the classic and oldest models and move up the ladder to the add-in approach. This excludes the latest addition to the family, the SharePoint Framework (SPFx), which the book will cover in deep detail in the following chapters.

We feel it's crucial to understand how things were built for previous (and sometimes current) versions of SharePoint, even if at a pretty high level, in order to respect and understand the decision and options we have at our disposal today.

Solutions for SharePoint and SharePoint Online

Solutions for SharePoint (primarily versions from 2007 to 2016) and SharePoint Online can be built in numerous ways. Next, we'll go through a bit of history because we feel it is important to have a glimpse of how (bad) things were in the beginning. It is often helpful to understand why things work in a certain way with SharePoint, as many technical functionalities in SharePoint are based on the pre-SharePoint Online era.

SharePoint 2001-2003: direct modification of files

As the title of this section states, this relates to the early days of SharePoint, between SharePoint Portal Server 2001 and SharePoint Portal Server 2003. These two versions did not have a way to formally package, deploy, or retract custom code. Developers resorted to all kinds of hacks and kludges that kind of work, but did not really respect any solid development approaches, such as application lifecycle management or version control.

Developers would simply open a remote connection (typically via Remote Desktop) to a SharePoint 2001/2003 server, find the file they needed to modify, and use either Notepad on the server or copy the file for additional editing on their own workstation. Files would reside in the SharePoint Hive, under such directories as C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\<version>\. Each SharePoint server, if you had several in a farm configuration, would have identical files. Thus, changing one file on a single server would require you to change the same file identically on other servers as well.

In a way, this was a quick, fast, and reasonably well-understood approach. If you needed a change within SharePoint, you could simply figure out which file was responsible for the change and inject your changes to that file only. The result would often be so-so and kind of acceptable--until the next Service Pack was released and installed on the very same SharePoint server. At best, it would simply wipe customizations and force you to re-do all your changes once more. At worst, it would break something horribly as some files would now be patched with the Service Pack's version, while others would be customized and there would be no knowing what had gone wrong and where. Pages wouldn't load and random errors would occur for users.

One inventive, although certainly unsupported approach, was to tinker with Windows' Internet Information Services (IIS) web server settings to fool SharePoint just a tiny bit. SharePoint typically, at the time, mapped the infamous /_layouts/ virtual directory to the SharePoint Hive under C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\<version>\ subdirectories. Clever system administrators and developers would copy the hive directory structure to a sibling directory structure, and map the virtual directory from /_layouts/ in IIS to this new copy of the original files. In practical terms, this allowed one to tinker with the files without fear that original files would be destroyed. Upon new Service Pack deployments, you could point the virtual directory back to the original hive, allow patching to complete, and then do a file diff comparison between the original (patched) hive and your customized hive. Talk about spending a lot of time to get to your results!

Luckily, SharePoint versions 2001 and 2003 are not supported and we haven't seen these versions in production in years. But this is where it all started back in the day.

SharePoint 2007--Full-trust code

Microsoft Office SharePoint Server 2007, or MOSS as it was known then, introduced the first true and quite proper development approach to SharePoint. This vehicle became known as SharePoint solution packages, that packaged our native .NET code to files with a .WSP extension.

This was also known as the full-trust code, as all code you wrote, packaged, deployed and used would run as fully trusted within SharePoint's memory space. A poor developer could easily write horribly sloppy code that iterated through hundreds of SharePoint sites recursively, then iterated through each list and document library and changed something in one or two of the document libraries. This would then be embedded in a web part, that is a building block that acts as a widget on a SharePoint page. Initially, the developer might have tested this on his or her laptop, and everything could have executed flawlessly. Change this to a production environment, with possibly hundreds or thousands of people loading the very same page, and you quickly run into all sorts of unforeseen performance issues.

Full-trust code initially did not have proper support within Visual Studio at the time, so developers would build custom tooling to compile, define, and package the deployment packages. Deployment would also be part of custom scripts and tooling executed locally on one or more SharePoint servers. Debugging would then occur on a separate server and would often be time-consuming and error-prone.

Solutions would be crafted around Features, which acted as a kind of lights on/lights off containment model for SharePoint artifacts. One or more Features would be packaged within a SharePoint solution package (the .WSP file) and uploaded to SharePoint's database for deployment. Each Feature would consist of at least one XML file introducing the deployment logic, and each solution package would consist of exactly one manifest file that held everything together. Combine this with possibly multiple solution packages per the overall solution and you often had a time-sensitive and highly detailed deployment script that needed to be executed just so, in the hope that everything would work.

What's important to realize is that MOSS 2007 gave us the Feature framework and the Solution framework. Much of the development between SharePoint versions 2007 and 2010 was tinkering and fiddling with XML file structures, essentially describing in pre-defined XML structures how a certain feature might act within SharePoint. Here and there developers could inject a little bit of HTML, combined with JavaScript, CSS and, if lucky, possibly also some C# code. We still remember countless meetings and review discussions with talented developers who swore never to work with SharePoint again, as it was not real coding, but would rather modify cumbersome XML structures that at times were poorly documented and lacked a strongly typed approach.

You wouldn't have compile-time error handling, and deployment to SharePoint might have taken 10 or 15 minutes for each try.

SharePoint solution packages (and Features) are still supported for all SharePoint versions from 2007, 2010 and 2013 to 2016. It's not advisable to continue building solution packages but, at times, a transformation from a legacy solution might be required. It should also be clear without saying, but just to be clear solution packages are not supported in SharePoint Online. As such, developers aiming to build cloud-supported solutions for SharePoint Online, should not build solution packages or solutions using the Feature framework.

SharePoint 2010 and SharePoint Online: sandbox solutions

With SharePoint 2010, Microsoft introduced a subset of the full-trust code development model. This was called sandbox solutions and, on paper, the idea was great. Not fabulous, but great.

The thinking at the time was that since organizations were starting to move more and more of their on-premises workloads to hosting environments or even public clouds (pre-Office 365 era), something needed to be done so that SharePoint deployments littered with full-trust code could safely be lifted and shifted or migrated to a multitenant platform. Since full-trust code was, well, fully trusted, it wouldn't take more than a few hours for someone to deploy bad-behaving code that would crash the whole application pool of a given IIS, and possibly even consume all available virtual memory within the SharePoint server.

Sandbox solutions were introduced for both SharePoint 2010 and SharePoint Online. These were initially meant for SharePoint 2010 and were offered as a multitenant platform for organizations so that they could share a common platform, and were later ported to SharePoint Online as a means to govern what kind of custom code would be deployed to a shared platform.

The idea with sandbox solutions was that running code within a package could only access a very limited set of APIs and namespaces within the SharePoint Object Model. One limitation that was set, meant that all calls to methods within the SharePoint development models could only access elements under a single site collection by using the SPSite() object. This way developers couldn't access any structures or data above the site collection that was running the code. Access to direct file I/O was also prohibited, as well as access to directly call HTTP interfaces through SharePoint.

This approach worked well, on paper. But in reality, porting existing code from full-trust packages proved to be troublesome, time-consuming, and sometimes just impossible. Existing code often resorted to using certain methods and features of the SharePoint Object Model, so that a replacement within the SPSite() object was not simply available. Therefore, either big pieces of custom code had to be cut in order to move the platform to the cloud, or fully rewritten without having proper APIs.

As part of this ideology, Microsoft introduced the idea of resources. Each call from the code would consume a small piece of resources from a pool of resources, and administrators could designate pre-allocations for each site collection. When the customization exceeded its allotted resources, it would simply be shut down. Imagine having a graphical component running on your SharePoint site that exposes a tool for your users. Suddenly, the tool simply disappears from the page because it exceeded a hidden resource allocation that end users weren't aware of. Administrators would have to reallocate more resources or start digging through code to understand where and how the resources were spent and how to best optimize this piece of code for a small subset of users complaining loudly.

As such, sandbox solutions never really took off. A couple of years after introducing sandbox solutions in SharePoint Online, Microsoft quietly deprecated the ability to run custom .NET code within a sandbox solution package. This effectively rendered the whole model to a vehicle for provisioning XML-based artifacts for a given SharePoint site or site collection. Not much changed, as developers were now complaining (again) that developing solutions for SharePoint were merely pointless exercises in understanding archaic XML structures in Notepad.

From time to time we still see customer deployment in SharePoint Online happily running sandbox solutions. This is fine but in no way advisable for the future. A good approach is to export the sandbox solution files, rename the packages from .WSP to .ZIP files, and explore what's in the package. This way you can learn and understand what the solution truly implement, and how to best replace the solution.

SharePoint 2013, SharePoint 2016, and SharePoint Online: add-ins

With SharePoint 2013, Microsoft felt it was time to introduce a new approach to partially (or fully) replace the now aging Feature and Solution framework development model. This model was initially launched to market with a catchy name, the Cloud App Model (CAM). As its name implies, CAM was crafted on purpose to provide a cloud-enabled development approach that fully supported SharePoint Online without the missteps of sandbox solutions. Later, CAM was renamed the App Model (AM) for a brief moment, and a while later it was renamed the Add-in model. For now, the Add-in model seems to have stuck, so in the context of development, we'll be referencing all types of apps as add-ins when they relate to SharePoint add-ins.

The winning ticket within the Add-in model was the de-coupling of server-side code from SharePoint. Developers were now able to code in their preferred programming language, the server-side implementation of their customization, and simply run that outside SharePoint. As such, it was never similar to full-trust code from the SharePoint 2007-2010 era, but more a model that allowed developers to execute code elsewhere.

This was also fairly close to the time when Microsoft Azure started gaining more and more ground and popularity within organizations. Ideally, companies could implement custom add-ins that would run in Microsoft Azure's Platform as a Service (PaaS) environment, and the surface on SharePoint with the help of the Add-in model's Client-Side Object Model (CSOM).

Add-ins initially came in three flavors:

• SharePoint-hosted Add-ins, which are add-ins without any server-side code or additional logic that would need to run outside SharePoint. They would use SharePoint's data structure, including a separate subsite within the current SharePoint site where the add-in was being provisioned. The subsite (called the AppWeb) would be used as a fixed storage for everything the add-in might need to record. The AppWeb could, obviously, store only data that fit the artifacts available for a SharePoint site: lists, document libraries, and similar quite limited elements. All user interface elements had to be built with CSS, HTML, and JavaScript, and all assets had to be hosted within the AppWeb to provide better security and in order to avoid cross-site scripting attacks.
• Provider-hosted add-ins, which are true add-ins in the sense that server-side code can be included as part of the add-in. The development consists of a separate website, that can be implemented with several supported frameworks and languages--even in non-Microsoft programming languages such as Python or Ruby on Rails. The idea is to implement a separate website and then call upon that site and embed the output of the site to a SharePoint site using a specifically crafted IFRame. For the most part, this solution turned out to be a fairly reasonable approach to running custom code outside SharePoint. For many scenarios though, performance was subpar, and security was problematic to configure and understand. The whole framework relied on carefully created public-key cryptography certificates that had to be deployed just in the right order in the hope of getting the platform to securely call the external website within the user's security realm. Provider-hosted add-ins are still built today, but they are not as common as one might expect. For several larger deployments with tens of thousands of users, we are seeing one or maybe two provider-hosted add-ins, and they tend to be rather complex implementations of internal applications that must play nicely with SharePoint's interface and look and feel.
• Auto-hosted add-ins were a novel idea to run add-ins in the cloud and if code required additional resources, they would be automatically provisioned from Microsoft Azure. There were too many questions on this approach--especially from customers who would ask which credit card or billing contract was used for the Azure-provisioned assets. Since then, Auto-hosted Add-ins have been retired and they have not supported or in use anymore.

Today, the Add-in model is still supported and alive. Not much progress has occurred with the model and it's mostly been stale since 2015.

SharePoint Online--add-ins and client-side scripts

In addition to add-ins, developers often need to drop small functionality and bits of features to individual pages within SharePoint sites. A common approach is to add a Script Editor Web Part on a page and drop a piece of JavaScript and/or HTML within the page. This is a very simple approach but can be both powerful and troublesome in the long run.

The benefits of using a Script Editor Web Part (SEWP) is that it's very easy to add as an ad-hoc solution when in a meeting with the site owner and simply code on-the-fly whatever is needed. This is assuming the developer is quite capable and fluent with JavaScript and the SharePoint APIs.

Adding a SEWP on a page allows developers to write JavaScript through the browser and save it into SharePoint Online's own database:

Upon page load, the script is executed and the user gets the result of the preceding code in a popup since the sample is using the classic JavaScript alert() message box:

The downside is that pages tend to become littered with all sorts of small JavaScript tidbits that no-one can keep track of. One piece of code could reference an external library that has a locally stored copy of a SharePoint document library, while another piece of code (even on the same page!) might reference the very same external library but a different version through a public CDN URI. Thus, the payload for a user loading the page would be at least double, as the client browser would need to resolve both external frameworks, load the payload, and figure out which version to use and when. Imagine having hundreds of sites, thousands of pages, with customizations using the SEWP-approach on five percent of all pages. You very quickly run into performance and supportability issues, as well as troubleshooting errors that are not obvious as frameworks are not referenced in a proper manner.

Development tooling for SharePoint Online

So far we've established that solutions built for SharePoint Online historically have been mostly about sandboxed solutions, custom scripts embedded on pages with the Script Editor Web Part, and custom add-ins that are either hosted within SharePoint or separately in Microsoft Azure (or a similar environment).

Developers use a reasonably new version of Visual Studio development tooling to build, package, and deploy their code to SharePoint Online. This can be Visual Studio 2015 or 2017, or newer when one becomes available. By installing Office Development Tools during Visual Studio installation, developers get to enjoy the pre-built templates for new projects that target SharePoint Online.

The only template that can be reasonably used today is the SharePoint Add-in template:

1. This template allows for the creation of a SharePoint add-in that can be either a SharePoint-hosted add-in or a Provider-hosted add-in. During the initial creation of the project, you can specify the target site for development, and this can be the developer site you created earlier within your SharePoint Online tenant:

2. Note that if you choose to create a provider-hosted add-in, you'll need to do additional configuration with certificates in order to secure the deployment. For a SharePoint-hosted add-in, no additional configuration is needed and you can start building your solution right away.
3. The solution consists of a single Feature, the Add-in package, and necessary support files and assets:

4. When deploying the solution, you can right-click on the project, and select Publish.

5. This allows you to publish the add-in as a package for manual deployment, or deploy the package to Office Store. Office Store is a marketplace run by Microsoft, allowing for reselling and marketing of your add-in to other organizations.
6. The packaged add-in can now be manually deployed to your developer site. First, simply upload the add-in package file to your developer site's Apps in Testing document library. Upon completion of the upload, you can simply deploy the app to the same site:

7. As the add-in is merely a sample add-in, we haven't introduced any additional user permissions we might be needing while impersonating the logged-in user (executing the add-in). As such, we can simply trust the add-in:

8. The add-in will now install and self-provision within the site:

9. Installation might take a minute or two. After it completes, the ribbon has a Launch AppButton that becomes enabled when the app is selected in the document library. Clicking the button will call the add-in:

10. Since we didn't configure the add-in at all and just compiled the default template, the add-in is a full page app. It inherits the SharePoint Online look and feel, but is running in a different base URL to the SharePoint Online site it was launched from:

11. The add-in calls a small piece of JavaScript to resolve the logged-in user's full name. Everything else is static on the page, including the top-most navigation. The same add-in could also be embedded in a SharePoint page if we add an App Part instruction to tell SharePoint Online it can embed the add-in safely.