This article is the sample showing Gosip custom auth with AAD Certificate Authorization.

Azure App registration

1. Create or use existing app registration

2. Make sure that the app is configured for a specific auth scenario:

• Certificate

Follow instructions: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

• O365 Admin -> Azure Active Directory

• Generate self-signed certificate

# PowerShell, run on a Windows machine
$certName = "MyCert"
$password = "MyPassword"

$startDate = Get-Date
$endDate = (Get-Date).AddYears(5)
$securePass = (ConvertTo-SecureString -String $password -AsPlainText -Force)

.\Create-SelfSignedCertificate.ps1 -CommonName $certName -StartDate $startDate -EndDate $endDate -Password $securePass

or on a Linux or macOS client via openssl:

chmod +x ./Create-SelfSignedCertificate.sh
./Create-SelfSignedCertificate.sh

• New App Registration

  • Accounts in this organizational directory only

  • API Permissions -> SharePoint :: Application :: Sites.FullControl.All -> Grant Admin Consent

  • Certificates & Secrets -> Upload .cer file

JSON

private.json sample:

{
	"siteUrl": "https://contoso.sharepoint.com/sites/test",
	"tenantId": "e4d43069-8ecb-49c4-8178-5bec83c53e9d",
	"clientId": "628cc712-c9a4-48f0-a059-af64bdbb4be5",
	"certPath": "cert.pfx",
	"certPass": "password"
}

Usage sample

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/azurecert"
)

func main() {

	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// 	TenantID: os.Getenv("AZURE_TENANT_ID"),
	// 	ClientID: os.Getenv("AZURE_CLIENT_ID"),
	// 	CertPath: os.Getenv("AZURE_CERTIFICATE_PATH"),
	// 	CertPass: os.Getenv("AZURE_CERTIFICATE_PASSWORD"),
	// }
	// or using `private.json` creds source

	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}
	
	client := &gosip.SPClient{AuthCnfg: authCnfg}
	sp := api.NewSP(client)

	res, err := sp.Web().Select("Title").Get()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Site title: %s\n", res.Data().Title)

}

Authentication strategies

Auth strategy should be selected corresponding to your SharePoint environment and its configuration.

Import path strategy "github.com/koltyakov/gosip/auth/{strategy}". Where /{strategy} stands for a strategy auth package.

JSON and struct representations are different in terms of language notations. So credentials parameters names in private.json files and declared as structs initiators vary.

Additional strategies

Gosip supports custom (ad hoc) strategies. Some worthy are boiled in the Sandbox to be added later on to the main package in a case of the demand.

Secrets encoding

When storing credential in local private.json files, which can be handy in local development scenarios, we strongly recommend to encode secrets such as password or clientSecret using cpass. Cpass converts a secret to an encrypted representation which can only be decrypted on the same machine where it was generated. This minimize incidental leaks, i.e. with git commits.

Supported strategies

Honorable mentions

Main features

• Unattended authentication using different strategies.

• Simplified API consumption (REST, CSOM, SOAP, etc.).

• SharePoint-aware embedded features (retries, header presets, error handling).

Supported SharePoint versions

• SharePoint Online (SPO).

• On-Premises (2019/2016/2013).

Supported auth strategies

• SharePoint Online:
• SharePoint On-Premises 2019/2016/2013:

Installation

Usage insights

Understand SharePoint environment type and authentication strategy

Let's assume it's SharePoint Online and Add-In Only permissions. Then strategy "github.com/koltyakov/gosip/auth/addin" sub package should be used.

Initiate authentication object

AuthCnfg's from different strategies contains different options relevant for a specified auth type.

Bind auth client with Fluent API

Usage samples

Fluent API client

Provides a simple way of constructing API endpoint calls with IntelliSense and chainable syntax.

Generic HTTP-client helper

Provides generic GET/POST helpers for REST operations, reducing amount of http.NewRequest scaffolded code, can be used for custom or not covered with a Fluent API endpoints.

Low-level HTTP-client usage

Low-lever SharePoint-aware HTTP client from github.com/koltyakov/gosip package for custom or not covered with a Fluent API client endpoints with granular control for HTTP request, response, and http.Client parameters. Used internally but almost never required in a consumer code.

SPClient has Execute method which is a wrapper function injecting SharePoint authentication and ending up calling http.Client's Do method.

Reference

License

This authentication option uses Microsoft Online Security Token Service https://login.microsoftonline.com/extSTS.srf and SAML tokens in order to obtain authentication cookie.

Struct

type AuthCnfg struct {
  // SPSite or SPWeb URL, which is the context target for the API calls
  SiteURL string `json:"siteUrl"`
  // Username for SharePoint Online, e.g. `[user]@[company].onmicrosoft.com`
  Username string `json:"username"`
  // User or App password
  Password string `json:"password"`
}

JSON

private.json sample:

{
  "siteUrl": "https://contoso.sharepoint.com/sites/test",
  "username": "john.doe@contoso.onmicrosoft.com",
  "password": "this-is-not-a-real-password"
}

Code sample

package main

import (
	"log"
	// "os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/saml"
)

func main() {
	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// 	Username: os.Getenv("SPAUTH_USERNAME"),
	// 	Password: os.Getenv("SPAUTH_PASSWORD"),
	// }
	// or using `private.json` creds source

	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...
}

The default NTLM authentication uses go-ntlmssp - the great package by Azure team. However, we found rare environments where it fails for some reason. Fortunately, there is an alternative.

Usage sample

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip-sandbox/strategies/ntlm"
)

func main() {

	authCnfg := &strategy.AuthCnfg{
		SiteURL:  os.Getenv("SPAUTH_SITEURL"),
		Username: os.Getenv("SPAUTH_USERNAME"),
    Domain:   os.Getenv("SPAUTH_DOMAIN"),
		Password: os.Getenv("SPAUTH_PASSWORD"),
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	sp := api.NewSP(client)

	res, err := sp.Web().Select("Title").Get()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Site title: %s\n", res.Data().Title)

}

Use the alternative only as a fallback method if go-ntlmssp doesn't work and don't forget to post an issue to help making Open Source tools better.

This article is the sample showing Gosip custom auth with AAD Device Token Authorization.

If you want users to sign in interactively, the best way is through device token authentication. This authentication flow passes the user a token to paste into a Microsoft sign-in site, where they then authenticate with an Azure Active Directory (AAD) account. This authentication method supports accounts that have multi-factor authentication enabled, unlike standard username/password authentication.

Azure App registration

1. Create or use existing app registration

2. Make sure that the app is configured to support device flow

• Authentication settings

  • Public client/native (mobile & desktop)

  • Suggested Redirect URIs for public clients (mobile, desktop) - https://login.microsoftonline.com/common/oauth2/nativeclient - checked

  • Default client type - Yes - for Device code flow, learn more

• App permissions

  • Azure Service Management :: user_impersonation

  • SharePoint :: based on your application requirements

• etc. based on application needs

Auth configuration and usage

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/device"
)

func main() {

	authCnfg := &strategy.AuthCnfg{
		SiteURL:  os.Getenv("SPAUTH_SITEURL"),
		ClientID: os.Getenv("SPAUTH_AAD_CLIENTID"),
		TenantID: os.Getenv("SPAUTH_AAD_TENANTID"),
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	sp := api.NewSP(client)

	res, err := sp.Web().Select("Title").Get()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Site title: %s\n", res.Data().Title)

}

When started the application interacts with user using device login.

To sign in, use a web browser to open the page https://microsoft.com/devicelogin 
and enter the code CL25ZF5N7 to authenticate.

After opening the link, providing device code and authenticating in browser the app is ready for communication with your SharePoint site.

The strategy caches auth token in the context of the AAD ClientID. As a result, you won't see the sign in message. If it's not the desired behavior .CleanTokenCache() method can be called to clean the local cache.

Note, that the technique is mostly applicable when user interaction is assumed. Usage of that auth approach in the headless scenarios is not the best as it can lead "stuck" application if no-one expects sign in interaction.

During the development, it's common to face a situation when production-level auth (AddIn Onli, Azure AD application) can't be configured in the desired timeframes and no auth strategies work. A simple example might be 2FA (multi-factor authentication) or custom ADFS provider. As a quick workaround, the On-Demand auth can help.

On-Demand means that an interactive browser session is started where a user can provide the credentials as if he/she opens the SharePoint site and follows the same flow as reaching the site in a browser.

In that strategy, the application actually opens the browser and communicates via debug protocol for the auth cookies when uses them in the requests.

On-Demand auth is based on Lorca project, however, a vital part of the functionality is not exposed as a public API in Lorca, so the dependency is imported from a fork with only that small change in exposing one additional method.

Lorca masters Chrome Debug Protocol, therefore, the Chrome/Chromium browser must be installed in the system where On-Demand auth is intended to be called.

Configure and usage sample

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip-sandbox/strategies/ondemand"
)

func main() {

	authCnfg := &strategy.AuthCnfg{
		SiteURL: os.Getenv("SPAUTH_SITEURL"),
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	sp := api.NewSP(client)

	res, err := sp.Web().Select("Title").Get()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Site title: %s\n", res.Data().Title)

}

On-Demand configuration assumes only SiteURL to be provided as everything else is dynamically resolved while the transition to the browser page.

The auth technique works for any strategy which is based on the cookies.

The strategy caches the cookies in the context of the SharePoint host. As a result, you won't see the credentials prompt each time. If it's not the desired behavior .CleanCookieCache() method can be called to clean the local cache.

Note, that the technique is only applicable when user interaction is assumed. Never ever use that auth approach in headless scenarios.

Azure App registration

1. Create or use existing app registration

2. Make sure that the app is configured for a specific auth scenario:

• Client credentials (might not work with SharePoint but require a Certificate-based auth)

• Certificate

• Username/Password (public clients flows must be enabled)

• Managed identity

Follow instructions: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

• O365 Admin -> Azure Active Directory

• Generate self-signed certificate

# PowerShell, run on a Windows machine
$certName = "MyCert"
$password = "MyPassword"

$startDate = Get-Date
$endDate = (Get-Date).AddYears(5)
$securePass = (ConvertTo-SecureString -String $password -AsPlainText -Force)

.\Create-SelfSignedCertificate.ps1 -CommonName $certName -StartDate $startDate -EndDate $endDate -Password $securePass

or on a Linux or macOS client via openssl:

chmod +x ./Create-SelfSignedCertificate.sh
./Create-SelfSignedCertificate.sh

• New App Registration

  • Accounts in this organizational directory only

  • API Permissions -> SharePoint :: Application :: Sites.FullControl.All -> Grant Admin Consent

  • Certificates & Secrets -> Upload .cer file

• Use environment variables to provide creds bindings:

  • AZURE_TENANT_ID - Directory (tenant) ID in App Registration

  • AZURE_CLIENT_ID - Application (client) ID in App Registration

  • For certificate-base auth:

    • AZURE_CERTIFICATE_PATH - path to .pfx file

    • AZURE_CERTIFICATE_PASSWORD - password used for self-signed certificate

  • For username/password auth:

    • AZURE_USERNAME

    • AZURE_PASSWORD

Auth configuration and usage

package main

import (
    "fmt"
    "log"
    "os"

    "github.com/koltyakov/gosip"
    "github.com/koltyakov/gosip/api"
    strategy "github.com/koltyakov/gosip-sandbox/strategies/azureenv"
)

func main() {

    // os.Setenv("AZURE_TENANT_ID", "b1bacba7-c38a-414b-8c8b-65df26a15749")
    // os.Setenv("AZURE_CLIENT_ID", "8ca10ce6-c3d5-47c6-b803-0ef3b619f464")
    // os.Setenv("AZURE_CERTIFICATE_PATH", "/path/to/cert.pfx")
    // os.Setenv("AZURE_CERTIFICATE_PASSWORD", "cert-password")

    authCnfg := &strategy.AuthCnfg{
        SiteURL: os.Getenv("SPAUTH_SITEURL"),
    }

    client := &gosip.SPClient{AuthCnfg: authCnfg}
    sp := api.NewSP(client)

    res, err := sp.Web().Select("Title").Get()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Site title: %s\n", res.Data().Title)

}

Environment variables auto-injection

Environment variables can be automatically injected in a runtime for Azure AAD library. To use injection add correcponding environment variables in private.json into env JSON property:

{
  "siteUrl": "https://contoso.sharepoint.com/sites/site",
  "env": {
    "AZURE_TENANT_ID": "1efde0dc-21f5-4d3d-a053-1da762c7838c",
    "AZURE_CLIENT_ID": "7278fe9b-acd5-4be5-b688-999603560d31",
    "AZURE_CERTIFICATE_PATH": "./certs/MyCert.pfx",
    "AZURE_CERTIFICATE_PASSWORD": "MyPassword"
  }
}

Why Golang?

Go has many strong parts. It's damn simple, friendly for large team projects, super fast in a compilation, fast in runtime and requires little or no prerequisites. It is friendly for concurrent processes, modern hardware architectures, network applications and cloud-based solutions, DevOps tool.

Go promises backward compatibility and long support period. Many say that Go is "the next enterprise big thing" and one of the default preferred languages. Lots of modern huge and awesome applications are written in Go while staying simple and reliable. Many main vendors use Go in one way or another.

Go has lots of wonderful libraries at disposal. But little or no for SharePoint... until Gosip. So it's our spot to provide such an option.

For whom is this for?

In the first place, Gosip is for developers who use Go on their daily basis and who got to integrate their apps with SharePoint without diving too deep to learn it from the grouds up.

SharePoint is complicated. We spent years solving different complex and misc problems with the platform. Not only for the end-users and customers but also for other developers and Open Source community.

Recently more and more of our internal tools have been written in Go. Our opinion is that a tool is better if it is used in a variety of different scenarios and by different teams. So we decided to deliver our Go+SP experience to Open Source and improve it together with you.

If you have no idea what is Go probably Gosip is not for you, but if you have no "Why Go?" question Gosip is an option to master SharePoint API with ease.

Where to use?

Anywhere where you considering an application written in Go reasonable.

Use-cases can vary, let me describe just a few we use the library intensively:

• Web services and APIs (dedicated web API consumed by many clients)

• Microservices applications

• Schedule runners, workflow workers, queue listeners

• CLI based tools & scripting

• Relays and gateways with elements of API communication

• Development toolchains

We would love to hear your unique use-case and even help you with our opinion.

Why not C#, PowerShell or Node.js

Under any circumstances, we're not even suggesting replacing anything with anything. It's just wrong positioning.

Gosip and Go, in particular, is yet another available option that might or might not suit one needs. Even more, we use all the combination of technologies together following common sense, expertise, constraints, and many more factors. Sometimes something in Go helps in a PowerShell script. Sometimes a great .Net library is used together in Go worker. And don't forget we're also sort of ambassadors of Node.js ecosystem for SharePoint.

Together is stronger. For no reason don't search for a holly war aspect use whatever feels better for you for delivering great product or service.

This article is the sample showing Gosip custom auth with AAD Username/Password Authorization.

Azure App registration

JSON

private.json sample:

{
	"siteUrl": "https://contoso.sharepoint.com/sites/test",
	"tenantId": "e4d43069-8ecb-49c4-8178-5bec83c53e9d",
	"clientId": "628cc712-c9a4-48f0-a059-af64bdbb4be5",
	"username": "user@contoso.com",
	"password": "password"
}

Usage sample

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/azurecreds"
)

func main() {

	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// 	TenantID: os.Getenv("AZURE_TENANT_ID"),
	// 	ClientID: os.Getenv("AZURE_CLIENT_ID"),
	// 	Username: os.Getenv("AZURE_USERNAME"),
	// 	Password: os.Getenv("AZURE_PASSWORD"),
	// }
	// or using `private.json` creds source

	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}
	
	client := &gosip.SPClient{AuthCnfg: authCnfg}
	sp := api.NewSP(client)

	res, err := sp.Web().Select("Title").Get()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Site title: %s\n", res.Data().Title)

}

Struct

type AuthCnfg struct {
  // SPSite or SPWeb URL, which is the context target for the API calls
  SiteURL string `json:"siteUrl"`
  // Client ID obtained when registering the AddIn
  ClientID string `json:"clientId"`
  // Client Secret obtained when registering the AddIn
  ClientSecret string `json:"clientSecret"`
  // Your SharePoint Online tenant ID (optional)
  Realm string `json:"realm"`
}

Realm can be left empty or filled in, that will add small performance improvement. The easiest way to find tenant is to open SharePoint Online site collection, click Site Settings -> Site App Permissions. Taking any random app, the tenant ID (realm) is the GUID part after the @.

See more details of AddIn Configuration and Permissions.

JSON

private.json sample:

{
  "siteUrl": "https://contoso.sharepoint.com/sites/test",
  "clientId": "e2763c6d-7ee6-41d6-b15c-dd1f75f90b8f",
  "clientSecret": "OqDSAAuBChzI+uOX0OUhXxiOYo1g6X7mjXCVA9mSF/0="
}

Code sample

package main

import (
	"log"
	// "os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/addin"
)

func main() {
	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:      os.Getenv("SPAUTH_SITEURL"),
	// 	ClientID:     os.Getenv("SPAUTH_CLIENTID"),
	// 	ClientSecret: os.Getenv("SPAUTH_CLIENTSECRET"),
	// }
	// or using `private.json` creds source

	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...
}

Extending client secrets

It's important to know that the legacy AddIn authentication's Client Secrets are issued for a limited time. After expiration, if not managed right way there is a risk to get a service connection aunothorized with the following message:

AADSTS7000222: The provided client secret keys for app '***' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.

To renew an AddIn please follow https://docs.microsoft.com/ru-ru/sharepoint/dev/sp-add-ins/replace-an-expiring-client-secret-in-a-sharepoint-add-in or, long story short:

Install-Module -Name AzureAD
Install-Module MSOnline

Connect-MsolService # provide tenant admin account creds

$clientId = 'e2763c6d-7ee6-41d6-b15c-dd1f75f90b8f' # replace with your clientId
$bytes = New-Object Byte[] 32
$rand = [System.Security.Cryptography.RandomNumberGenerator]::Create()
$rand.GetBytes($bytes)
$rand.Dispose()
$newClientSecret = [System.Convert]::ToBase64String($bytes)
New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Symmetric -Usage Sign -Value $newClientSecret -StartDate (Get-Date) -EndDate (Get-Date).AddYears(1)
New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Symmetric -Usage Verify -Value $newClientSecret -StartDate (Get-Date) -EndDate (Get-Date).AddYears(1)
New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Password -Usage Verify -Value $newClientSecret -StartDate (Get-Date) -EndDate (Get-Date).AddYears(1)
$newClientSecret # outputs new clientSecret

Known issues

AddIn Only auth is considered a legacy, in a production Azure Cert is vendor recommended.

In new subscriptions you can face Grant App Permission disabled. You'll be getting the following error:

{
  "error": "invalid_request",
  "error_description": "Token type is not allowed."
}

To enable this feature, connect to SharePoint using Windows PowerShell and then run:

set-spotenant -DisableCustomAppAuthentication $false.

Install-Module -Name Microsoft.Online.SharePoint.PowerShell  
$adminUPN="<the full email address of a SharePoint administrator account, example: jdoe@contosotoycompany.onmicrosoft.com>"  
$orgName="<name of your Office 365 organization, example: contosotoycompany>"  
$userCredential = Get-Credential -UserName $adminUPN -Message "Type the password."  
Connect-SPOService -Url https://$orgName-admin.sharepoint.com -Credential $userCredential  
set-spotenant -DisableCustomAppAuthentication $false  

It's not an auth strategy but a mode without any authentication flow applied to the SPClient.

Anonymous mode can be handy in a situation when Gosip SharePoint-aware helpers intended to be used however authentication is handled by any other middleware.

Struct

Only SiteURL is required.

type AuthCnfg struct {
	// SPSite or SPWeb URL, which is the context target for the API calls
	SiteURL string `json:"siteUrl"`
}

JSON

private.json sample:

{
  "siteUrl": "https://www.contoso.com/sites/test"
}

Code sample

package main

import (
	"log"
	// "os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/anon"
)

func main() {

	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// }
	// or using `private.json` creds source
	
	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...

}

For AddIn Only authentication to work register new addin within your SharePoint Online tenant.

• Navigate to app registration page: https://{organization}.sharepoint.com/sites/{site}/_layouts/15/appregnew.aspx

• Click "Generate" button next to Client Id and Client Secret, fill in Title, App Domain, Redirect URI (you can type in any values you want).

• Copy Client Id and Client Secret and press "Create" button.

• Apply permissions for the app on tenant or site collection level.

Tenant scoped parmissions

https://{organization}-admin.sharepoint.com/_layouts/15/appinv.aspx

Site collection scoped permissions

https://{organization}.sharepoint.com/sites/{site}/_layouts/15/appinv.aspx

• Resolve addin by Client Id and paste in App's Permissions Request XML:

• Click "Create" and "Trust It".

To check which app principals are assigned for a site collection use:

https://{organization}.sharepoint.com/sites/{site}/_layouts/15/appprincipals.aspx

Disabled by default

In new subscriptions you could be needed to enable Grant App Permission. Connect to SharePoint using Windows PowerShell and then run:

set-spotenant -DisableCustomAppAuthentication $false.

Currently is legacy but was a popular way of exposing SharePoint into external world back in the days.

Struct

type AuthCnfg struct {
  // SPSite or SPWeb URL, which is the context target for the API calls
  SiteURL string `json:"siteUrl"`
  // Username for SharePoint On-Prem,
  // format depends in TMG settings, can include domain or doesn't
  Username string `json:"username"`
  // User password
  Password string `json:"password"`
}

JSON

private.json sample:

{
  "siteUrl": "https://www.contoso.com/sites/test",
  "username": "john.doe",
  "password": "this-is-not-a-real-password"
}

Code sample

package main

import (
	"log"
	// "os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/tmg"
)

func main() {
	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// 	Username: os.Getenv("SPAUTH_USERNAME"),
	// 	Password: os.Getenv("SPAUTH_PASSWORD"),
	// }
	// or using `private.json` creds source
	
	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...
}

Binding authentication and API client

Understand SharePoint environment type and authentication strategy

Based on authentication provider supported and configured in your SharePoint Farm environment different library authentication strategies might be or not be applicable.

If you have no idea which strategy is used within your farm the question is better be addressed to SharePoint admins.

Let's assume it's SharePoint Online and Add-In Only permissions. Then strategy "github.com/koltyakov/gosip/auth/addin" sub package should be used.

package main

import (
  "github.com/koltyakov/gosip"
  "github.com/koltyakov/gosip/api"
  strategy "github.com/koltyakov/gosip/auth/addin"
)

It could have been SharePoint On-Premise and NTLM and strategy "github.com/koltyakov/gosip/auth/ntlm" as imported strategy.

Initiate authentication object

Different authentication strategies assumes different credential parameters. Sometimes it can be Username/Password, sometimes CertPath or ClientID/ClientSecret. Please refer a specific strategy documentation for relevalt for the auth type parameters.

Credential can be passed directly in AuthCnfg's struct. Here a sample for addin:

auth := &strategy.AuthCnfg{
  SiteURL:      os.Getenv("SPAUTH_SITEURL"),
  ClientID:     os.Getenv("SPAUTH_CLIENTID"),
  ClientSecret: os.Getenv("SPAUTH_CLIENTSECRET"),
}

An anternative is using a configuration file (see more). Which is a JSON containing the same parameters as used with AuthCnfg's struct.

configPath := "./config/private.json"
auth := &strategy.AuthCnfg{}

err := auth.ReadConfig(configPath)
if err != nil {
  fmt.Printf("Unable to get config: %v\n", err)
  return
}

Bind auth client with Fluent API

Now when auth is bound it should be passed to client and Fluent API instance:

client := &gosip.SPClient{AuthCnfg: auth}

sp := api.NewSP(client)

Most of the samples starts with sp assuming configuration described above is already in place.

res, err := sp.Web().Select("Title").Get()
if err != nil {
  fmt.Println(err)
}

fmt.Printf("%s\n", res.Data().Title)

Gosip HTTP client for SharePoint allows consuming any HTTP resource or API, it can even bypass SharePoint site pages (with all assets) within a dev toolchain (proxy, dev server).

Gosip HTTP client is SharePoint nuances-aware, it takes care under the hood of such things as headers, API calls retries, threshholds, error handling, POST API requests Digests, and, of course, authentication and its renewal.

However, dealing at low-level, means you should know SharePoint API rather well and you're ok with the verbosity of Go. If you just starting with SharePoint please consider Fluent API client first and HTTP client for none covered methods and custom stuff.

Usage

package main

import (
	"fmt"
	"log"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/ntlm"
)

func main() {
	auth := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := auth.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v\n", err)
	}

	spClient := api.NewHTTPClient(&gosip.SPClient{AuthCnfg: auth})

	endpoint := auth.GetSiteURL() + "/_api/web?$select=Title"

	data, err := spClient.Get(endpoint, nil)
	if err != nil {
			log.Fatalf("%v\n", err)
	}

	// spClient.Post(endpoint, body, nil) // generic POST

	// generic DELETE helper crafts "X-Http-Method"="DELETE" header
	// spClient.Delete(endpoint, nil)

	// generic UPDATE helper crafts "X-Http-Method"="MERGE" header
	// spClient.Update(endpoint, body, nil)

	// CSOM helper (client.svc/ProcessQuery)
	// spClient.ProcessQuery(endpoint, body, nil)

	fmt.Printf("response: %s\n", data)
}

Methods

HTTP Client methods covers those which used in SharePoint all the way.

In a contract to default Go http.Client's .Do method, Gosip HTTP Client's methods proceed and read response body to return array of bytes. Which reduce amount of scaffolded code a bit and more handy for the REST API consumption, in our opinion, which is 90% of use cases.

Get

Sends GET request, embeds "Accept": "application/json;odata=verbose" header as a default. Use it for REST API GET calls.

Post

Sends POST request, embeds "Accept": "application/json;odata=verbose" and "Content-Type": "application/json;odata=verbose;charset=utf-8" headers as default. X-RequestDigest is received, cached, and embed automattically. Use it for REST API POST calls.

Delete

Sends POST request, embeds "Accept": "application/json;odata=verbose", "Content-Type": "application/json;odata=verbose;charset=utf-8", and "X-Http-Method": "DELETE" headers as default. X-RequestDigest is received, cached, and embed automattically. Use it for REST API POST calls with delete resource intention.

Update

Sends POST request, embeds "Accept": "application/json;odata=verbose", "Content-Type": "application/json;odata=verbose;charset=utf-8", and "X-Http-Method": "MERGE" headers as default. X-RequestDigest is received, cached, and embed automattically. Use it for REST API POST calls with update resource intention.

ProcessQuery

Sends POST request to /_vti_bin/client.svc/ProcessQuery endpoint (CSOM). All required headers for a CSOM request are embed automatically. Method's body should stand for a valid CSOM XML package. The response body is parsed for error handling, yet returned in it's original form.

Low-level HTTP client

Sometimes more control is required, e.g. when downloading files or large responses you could prefer precessing a response in chunks. This can be achieved with the low-level usage of Gosip HTTP client.

client := &gosip.SPClient{AuthCnfg: auth}

var req *http.Request
// Initiate API request
// ...

resp, err := client.Execute(req)
if err != nil {
  fmt.Printf("Unable to request api: %v", err)
  return
}

SPClient has Execute method which is a wrapper function injecting SharePoint authentication and ending up calling http.Client's Do method.

So you can dive down to native *http.Request at this point it's just a standard request from "net/http" package, but authenticated to SharePoint and with some batteries under the hood for a seemles API integration.

CRUD is the most commonly requested type of API consumption.

This example demonstrate basic operations on a list items sample. Hovewer, SharePoint has a variety of nuances even when it comes to just getting items from a list.

Getting list object

// The recommended way of getting lists is by using their relative URIs
// can be a short form without full web relative URL prefix
list := sp.Web().GetList("Lists/MyList")

// other common but less recommended way of getting a list is
// list := sp.Web().Lists().GetByTitle("My List")

Getting items

// itemsResp is a byte array read from response body
// powered with some processing "batteries" as Data() or Unmarshal() methods
itemsResp, err := list.Items().
	Select("Id,Title").  // OData $select modifier, limit what props are retrieved
	OrderBy("Id", true). // OData $orderby modifier, defines sort order
	Top(10).             // OData $top modifier, limits page size
	Get()                // Finalizes API constructor and sends a response

if err != nil {
	log.Fatal(err)
}

// Data() method is a helper which unmarshals generic structure
// use custom structs and unmarshal for custom fields
for _, item := range itemsResp.Data() {
	itemData := item.Data()
	fmt.Printf("ID: %d, Title: %s\n", itemData.ID, itemData.Title)
}

Based on a use case, items can be reveived with alternative methods, as .GetAll, .GetPaged, .GetByCAML, or even lists methods as .RenderListData.

Adding an item

// Payload should be a valid JSON stringified string converted to byte array
// Payload's properties must be a valid OData entity types
itemPayload := []byte(`{
	"Title": "New Item"
}`)

// Constructs and sends add operation request
// itemAddRes is a byte array read from response body with extra methods
itemAddRes, err := list.Items().Add(itemPayload)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Raw response: %s\n", itemAddRes)
fmt.Printf("Added item's ID: %d\n", itemAddRes.Data().ID)

Payloads can be constructed in a usual Go way using marshalling struct or string maps to JSON. E.g.:

itemMetadata := &struct{
	Title string `json:"Title"`
}{
	Title: "New Item",
}

itemPayload, _ := json.Marshal(itemMetadata)

or:

itemMetadata := map[string]interface{}{
	"Title": "New Item",
}

itemPayload, _ := json.Marshal(itemMetadata)

up to your preferences.

Getting a specific item

itemID := 42 // should specific item ID

itemRes, err := list.Items().GetByID(itemID).Get()
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Raw response: %s\n", itemRes)
fmt.Printf("Item metadata: %+v\n", itemRes.Data())

Updating an item

// Payload should be a valid JSON stringified string converted to byte array
// Payload's properties must be a valid OData entity types
itemUpdatePayload := []byte(`{
	"Title": "Updated Title"
}`)

itemID := 42 // should specific item ID

// Constructs and sends update operation request
// itemUpdateRes is a byte array read from response body with extra methods
itemUpdateRes, err := list.Items().GetById(itemID).Update(itemUpdatePayload)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Item is updated, %s\n", itemUpdateRes.Data().Modified)

Payloads can be constructed in a usual Go way using marshalling struct or string maps to JSON, same as in add operations.

Delete an item

Item can be not only deleted but recycled with a further restore operation, which can be provide more safety.

itemID := 42 // should specific item ID

// list.Items().GetByID(itemID).Delete() // or .Recycle()
if _, err := list.Items().GetByID(itemID).Recycle(); err != nil {
	log.Fatal(err)
}

Provides a simple way of constructing API endpoint calls with IntelliSense and chainable syntax.

Usage sample

package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/addin"
)

func main() {
	// Getting auth params and client
	client, err := getAuthClient()
	if err != nil {
		log.Fatalln(err)
	}

	// Binding SharePoint API
	sp := api.NewSP(client)

	// Custom headers (optional)
	headers := map[string]string{
		"Accept": "application/json;odata=minimalmetadata",
		"Accept-Language": "de-DE,de;q=0.9",
	}
	config := &api.RequestConfig{Headers: headers}

	// Chainable request sample
	data, err := sp.Conf(config).Web().Lists().Select("Id,Title").Get()
	if err != nil {
		log.Fatalln(err)
	}

	// Response object unmarshalling
	// struct depends on OData mode and API method
	res := &struct {
		Value []struct {
			ID    string `json:"Id"`
			Title string `json:"Title"`
		} `json:"value"`
	}{}

	if err := json.Unmarshal(data, &res); err != nil {
		log.Fatalf("unable to parse the response: %v", err)
	}

	for _, list := range res.Value {
		fmt.Printf("%+v\n", list)
	}
}

func getAuthClient() (*gosip.SPClient, error) {
	configPath := "./config/private.spo-addin.json"
	auth := &strategy.AuthCnfg{}
	if err := auth.ReadConfig(configPath); err != nil {
		return nil, fmt.Errorf("unable to get config: %v", err)
	}
	return &gosip.SPClient{AuthCnfg: auth}, nil
}

Main concepts

• Get authenticated

• Construct root SP object using api.NewSP(client)

• Construct API calls in a fluent way

• Parse responses in the Go way

• Embrase strongly typed generic responses

• Build awesome apps in Go for SharePoint

Gosip provides an events system with a set of handlers that can be optionally defined to track different client communication aspects such as request tracking, retries and response error logging, to name just a few.

To define the handlers Hooks object should be configured and passed to gosip.SPClient struct.

var authCnfg gosip.AuthCnfg
// ... auth config initiation is omitted

&gosip.SPClient{
  AuthCnfg: authCnfg,
  Hooks:    &gosip.HookHandlers{
    // handlers function definition
  },
}

The following handlers are available at the moment:

// HookHandlers struct to configure events handlers
type HookHandlers struct {
	OnError    func(event *HookEvent) // when error appeared
	OnRetry    func(event *HookEvent) // before retry request
	OnRequest  func(event *HookEvent) // before request is sent
	OnResponse func(event *HookEvent) // after response is received
}

All of the handlers are optional.

A handler receives HookEvent pointer which contains request pointer, response status code, and error (if applicable for an event), and time information to track duration since a request started an event happened.

Hooks sample:

// Define requests hook handlers
client.Hooks = &gosip.HookHandlers{
	OnError: func(e *gosip.HookEvent) {
		fmt.Println("\n======= On Error ========")
		fmt.Printf(" URL: %s\n", e.Request.URL)
		fmt.Printf(" StatusCode: %d\n", e.StatusCode)
		fmt.Printf(" Error: %s\n", e.Error)
		fmt.Printf("  took %f seconds\n",
		  time.Since(e.StartedAt).Seconds())
		fmt.Printf("=========================\n\n")
	},
	OnRetry: func(e *gosip.HookEvent) {
		fmt.Println("\n======= On Retry ========")
		fmt.Printf(" URL: %s\n", e.Request.URL)
		fmt.Printf(" StatusCode: %d\n", e.StatusCode)
		fmt.Printf(" Error: %s\n", e.Error)
		fmt.Printf("  took %f seconds\n",
		  time.Since(e.StartedAt).Seconds())
		fmt.Printf("=========================\n\n")
	},
	OnRequest: func(e *gosip.HookEvent) {
		if e.Error == nil {
			fmt.Println("\n====== On Request =======")
			fmt.Printf(" URL: %s\n", e.Request.URL)
			fmt.Printf("  auth injection took %f seconds\n",
			  time.Since(e.StartedAt).Seconds())
			fmt.Printf("=========================\n\n")
		}
	},
	OnResponse: func(e *gosip.HookEvent) {
		if e.Error == nil {
			fmt.Println("\n====== On Response =======")
			fmt.Printf(" URL: %s\n", e.Request.URL)
			fmt.Printf(" StatusCode: %d\n", e.StatusCode)
			fmt.Printf("  took %f seconds\n",
			  time.Since(e.StartedAt).Seconds())
			fmt.Printf("==========================\n\n")
		}
	},
}

Hooks can be handy for global logging streaming and metrics collection.

It is recommended using asynchronous and only lightweight logic inside hooks.

This type of authentication uses HTTP NTLM handshake in order to obtain authentication header.

Struct

type AuthCnfg struct {
  // SPSite or SPWeb URL, which is the context target for the API calls
  SiteURL  string `json:"siteUrl"`
  Domain   string `json:"domain"`   // AD domain name
  Username string `json:"username"` // AD user name
  Password string `json:"password"` // AD user password
}

Gosip uses github.com/Azure/go-ntlmssp NTLM negotiator, however a custom one also can be provided in case of demand.

JSON

private.json sample:

{
  "siteUrl": "https://www.contoso.com/sites/test",
  "username": "contoso\\john.doe",
  "password": "this-is-not-a-real-password"
}

or

{
  "siteUrl": "https://www.contoso.com/sites/test",
  "username": "john.doe",
  "domain": "contoso",
  "password": "this-is-not-a-real-password"
}

Code sample

package main

import (
	"log"
	// "os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/ntlm"
)

func main() {
	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// 	Username: os.Getenv("SPAUTH_USERNAME"),
	// 	Password: os.Getenv("SPAUTH_PASSWORD"),
	// }
	// or using `private.json` creds source

	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...
}

If this strategy doesn't work in your environment yet you know for sure it's NTLM used try this alternative.

Struct

type AuthCnfg struct {
  // SPSite or SPWeb URL, which is the context target for the API calls
  SiteURL string `json:"siteUrl"`
  // Username for SharePoint On-Prem,
  // format depends in FBA settings, can include domain or doesn't
  Username string `json:"username"`
  // User password
  Password string `json:"password"`
}

JSON

private.json sample:

{
  "siteUrl": "https://www.contoso.com/sites/test",
  "username": "john.doe",
  "password": "this-is-not-a-real-password"
}

Code sample

package main

import (
	"log"
	// "os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/fba"
)

func main() {
	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// 	Username: os.Getenv("SPAUTH_USERNAME"),
	// 	Password: os.Getenv("SPAUTH_PASSWORD"),
	// }
	// or using `private.json` creds source

	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...
}

Struct

type AuthCnfg struct {
  // SPSite or SPWeb URL, which is the context target for the API calls
  SiteURL      string `json:"siteUrl"`
  Username     string `json:"username"`
  Password     string `json:"password"`
  // Following are not required for SPO
  Domain       string `json:"domain"`
  RelyingParty string `json:"relyingParty"`
  AdfsURL      string `json:"adfsUrl"`
  AdfsCookie   string `json:"adfsCookie"`
}

See more details ADFS user credentials authentication.

Gosip's ADFS also supports a scenario of ADFS or NTML behind WAP (Web Application Proxy) which adds additional auth flow and EdgeAccessCookie involved into play.

JSON

On-Premises configuration

private.json sample:

{
  "siteUrl": "https://www.contoso.com/sites/test",
  "username": "john.doe@contoso.com",
  "password": "this-is-not-a-real-password",
  "relyingParty": "urn:sharepoint:www",
  "adfsUrl": "https://login.contoso.com",
  "adfsCookie": "FedAuth"
}

On-Premises behing WAP configuration

private.json sample:

{
  "siteUrl": "https://www.contoso.com/sites/test",
  "username": "john.doe@contoso.com",
  "password": "this-is-not-a-real-password",
  "relyingParty": "urn:AppProxy:com",
  "adfsUrl": "https://login.contoso.com",
  "adfsCookie": "EdgeAccessCookie"
}

SharePoint Online configuration

private.json sample:

{
  "siteUrl": "https://contoso.sharepoint.com/sites/test",
  "username": "john.doe@contoso.onmicrosoft.com",
  "password": "this-is-not-a-real-password"
}

Code sample

package main

import (
	"log"
	// "os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/adfs"
)

func main() {
	// authCnfg := &strategy.AuthCnfg{
	// 	SiteURL:  os.Getenv("SPAUTH_SITEURL"),
	// 	Username: os.Getenv("SPAUTH_USERNAME"),
	// 	Password: os.Getenv("SPAUTH_PASSWORD"),
	//	/* other auth props ...*/
	// }
	// or using `private.json` creds source

	authCnfg := &strategy.AuthCnfg{}
	configPath := "./config/private.json"
	if err := authCnfg.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v", err)
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...
}

Gosip HTTP client is preconfigured with retry policies so different kinds of failures can be back off. Current implementation of policies assumes a specific number of retries for a specific response HTTP Status Code. One would only consider retries for "non-ok" status codes and only those which represents server or networking error which has chances for backing off on a next try.

Defaults

The default policies are:

Retries' delay increases with each attempt: 200ms, 400ms, 800ms, 1.6s, 3.2s, and so on using the following progression formula:

time.Duration(100*math.Pow(2, float64(retry))) * time.Millisecond

For the responses with Retry-After header, retry after value is used in preference as threshold is usually require longer wait until next request permitted.

Custom policy

A custom policy can be provided on demand in the following way:

var authCnfg gosip.AuthCnfg
// ... auth config initiation is omitted

&gosip.SPClient{
	AuthCnfg: authCnfg,
	RetryPolicies: map[int]int{
		// merged with default policies
		500: 2,  // overwrites default
		503: 5,  // overwrites default
		401: 0,  // disables default
	},
}

Disabling retries for a specific request

When a specific request assumes no retries, e.g. resolving a folder which potentially doesn't exist but you know that 500 is returned in case of failure and it's a faster algorithm to try optimistically and check or create on an error only, it can be handy disabling any retries but only scoped to a specific request or series of requests.

For that purposes X-Gosip-NoRetry header is the resolution.

headers := map[string]string{
  "X-Gosip-NoRetry": "true",
}
conf := &RequestConfig{
  Headers: headers,
}
if _, err := sp.Web().GetFolder(folderURI).Conf(conf).Get(); err != nil {
  fmt.Println(err)
}

Custom retry delays

There is no direct way to redefine delays between retries, there was no such demand. However, extending delays is achievable via OnRetry hook. The OnRetry hook is called after it's known that a retry is needed but before the actual retry. The mechanics of retry checks is implemented in the way that should retry check actually also waits for the delay. In other words, OnRetry hook is fired right before the next retry is sent. So, if that hook is locked for a time the delay is increased correspondingly. That might be helpful for adding extra delay, yet won't work for the cases when the delays should be reduced or flatten.

While adding timeouts in hooks, make sure you're using something respecting context cancelation:

sleepTimeout := 1 * time.Second

select {
case <-e.Request.Context().Done():
  return
case <-time.After(sleepTimeout):
}

Gosip allows providing custom authentication mechanisms. For example, you are considering reusing Fluent API helpers and HTTP Client but existing authentication strategies do not feet your environment specifics. Maybe your tenant configured with custom ADFS provider, maybe it's 2FA and there are no alternatives and you need On-Demand auth, but it missed in Gosip strategies list? Fortunately, this is not any sort of stopper. All included authentication strategies are a sort of a pluging and it's rather affordable to add a new strategy on your own.

Let's take a look at any strategy binding:

package main

import (
	"log"
	"os"

	"github.com/koltyakov/gosip"
	strategy "github.com/koltyakov/gosip/auth/saml"
)

func main() {

	authCnfg := &strategy.AuthCnfg{
		SiteURL:  os.Getenv("SPAUTH_SITEURL"),
		Username: os.Getenv("SPAUTH_USERNAME"),
 		Password: os.Getenv("SPAUTH_PASSWORD"),
	}

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	// use client in raw requests or bind it with Fluent API ...

}

What we can see? Some strategy is imported into the strategy namespace. A strategy has AuthCnfg struct with some public properties which are obviously taking place in authentication flow. This struct is then passed to &gosip.SPClient{AuthCnfg: authCnfg} and somehow after following binding the requests are authenticated.

For this construction to work strategy.AuthCnfg should implement gosip.AuthCnfg interface which is:

// AuthCnfg is an abstract auth config interface,
// allows different authentications strategies' dependency injection
type AuthCnfg interface {
	// Authentication middleware fabric
	// applyes round tripper or enriches requests with authentication and metadata
	SetAuth(req *http.Request, client *SPClient) error

	// Authentication initializer (token/cookie/header, expiration, error)
	// to support cabability for exposing tokens for external tools
	// e.g. as of this sample project https://github.com/koltyakov/spvault
	GetAuth() (string, int64, error)

	ParseConfig(jsonConf []byte) error  // Parses credentials from a provided JSON byte array content
	ReadConfig(configPath string) error // Reads credentials from storage

	GetSiteURL() string  // SiteURL getter method
	GetStrategy() string // Strategy code getter
}

Philosophy of the strategies is to have two initiation modes, the first is a strict declaration of the creds and the second one is reading credentials from the config. That config is not necessarily a file on the file system it can be a request to a key vault or OS credential manager, etc.

As the interface is passed to gosip.SPClient struct, Gosip knows nothing about the creds and the context, for that reason GetSiteURL method is vital to target requests to a correct root URL.

GetStrategy method should return the string alias value of the strategy name if something specific should be happening based on its value.

GetAuth method is for token and cookie-based authentications, it can be omitted and return just a blank value, or it can be an actual place for authentication flow happening inside, returning a cached string which is when applied somehow to the requests making them authenticated. In case of custom logic, we'd recommend using GetAuth method and don't forget TTL caching to reduce roundtrips. With a robust external auth client, GetAuth can be dummy minimum (NTLM example shows this approach).

And finally, SetAuth, the method where all the magic happening. SetAuth method is a middleware, it receives runtime request and should append authentication stuff. Check these as samples: NTML's SetAuth, cookie-based auth, Bearer token-based auth.

By implementing AuthCnfg struct and gosip.AuthCnfg interface any custom authentication can be added to Gosip.

Our common recommendation with attachments is to use them in moderation with a preference to documents in libraries and linking business objects items to that document using metadata and other logical relationship. But sometimes you need nothing more than just a simple binary addition to an item.

Working with attachments is mostly straightforward as you can only get a list of item's attachment, get a specific attachment by its name, add and delete an attachment.

Getting attachments

list := sp.Web().GetList("Lists/MyList")
item := list.Items().GetByID(1)

attachments, err := item.Attachments().Get()
if err != nil {
	log.Fatal(err)
}

for _, attachment := range attachments.Data() {
	data := attachment.Data()
	fmt.Printf("%s (%s)\n", data.FileName, data.ServerRelativeURL)
}

Attachments API provides little information, actually only FileName and ServerRelativeURL.

Items have attachments

To detect which items have attachments the corresponding Attachments property can be requested within an ordinary get items request:

items, err := list.Items().Select("Id,Attachments").Get()
if err != nil {
	log.Fatal(err)
}

for _, item := range items.Data() {
	data := item.Data()
	hasAttachments := "has no"
	if data.Attachments {
		hasAttachments = "has"
	}
	fmt.Printf("Item ID %d %s attachments\n", data.ID, hasAttachments)
}

Adding attachments

Adding attachments is almost identical to adding documents to a document library.

list := sp.Web().GetList("Lists/MyList")
item := list.Items().GetByID(1)

content := strings.NewReader("Get content in a usual Go way you like")

item.Attachments().GetByName("MyAttachment.txt").Delete()

resp, err := item.Attachments().Add("MyAttachment.txt", content)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Attachment is added: %s\n", resp.Data().ServerRelativeURL)

Getting attachment by name

list := sp.Web().GetList("Lists/MyList")
item := list.Items().GetByID(1)

content, err := item.Attachments().GetByName("MyAttachment.txt").Download()
if err != nil {
	log.Fatal(err)
}

fmt.Printf(
	"Do whatever needed with this content of %d bytes\n",
	len(content),
)

Attachment actions

With an attachment you can:

• Download

• Get reader (download in a stream way)

• Delete

• or Recycle

These actions are rather obvious with the help of the Fluent API.

When you deal with multiple SharePoint environments with different strategies and concidering same code automatically resolving a strategy use the dynamic auth helper.

With the dynamic auth you'd need extending a private.json content with strategy property, containing a name of a strategy. E.g.:

{
  "strtegy": "addin",
  "siteUrl": "https://contoso.sharepoint.com/sites/site",
  "clientId": "...",
  "clientSecret": "..."
}

Available strategy names are: azurecert, azurecreds, device, addin, adfs, fba, ntlm, saml, tmg.

Usage

package main

import (
	"flag"
	"log"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	"github.com/koltyakov/gosip/auth"
)

func main() {
	config := flag.String("config", "./config/private.json", "Config path")

	flag.Parse()

	authCnfg, err := auth.NewAuthFromFile(*config)
	if err != nil {
		log.Fatalf("unable to get config: %v", err)
	}
	
	// or
	// authCnfg, _ := NewAuthFromFile(strategyName)
	// _ = auth.ParseConfig(byteJsonCreds)

	client := &gosip.SPClient{AuthCnfg: authCnfg}
	sp := api.NewSP(client)

	// ...

}

With a custom auth involved it can be extended like this.

Property bags are a nice way of storing global metadata and settings in SharePoint. Property bags are key-value pairs scoped to the container. In general, any folder can act as a container, also webs have their own property bag storage located in all properties section.

Use-cases can vary depending on how an application uses this key-value storage. For example, PnP Provisioning engine stores applied schema information in property bags. A good thing is that you don't need to provision any additional artifacts to keep some business logic or state variables.

A thing to know that all the props values are strings, dealing with different data types they should be serialized or converted to a string.

Getting web's all properties

props, err := sp.Web().AllProps().Get()
if err != nil {
	log.Fatal(err)
}

for key, val := range props.Data() {
	fmt.Printf("%s: %s\n", key, val)
}

Getting specific properties

To get a limited subset of properties .GetProps helper method can be used:

props, err := sp.Web().AllProps().GetProps([]string{
	"taxonomyhiddenlist",
	"vti_defaultlanguage",
})

if err != nil {
	log.Fatal(err)
}

for key, val := range props {
	fmt.Printf("%s: %s\n", key, val)
}

Selecting specific properties didn't work in old versions of SharePoint, however, the method has a fallback to getting all and filtering props on the client.

Setting property value

if err := sp.Web().AllProps().Set("my-prop", "my value"); err != nil {
	log.Fatal(err)
}

Modern SharePoint sites by default have custom scripting disabled mode. When custom scripting is disabled even an admin account will receive "Access denied. You do not have permission to perform this action or access this resource." error message. This is the expected behavior.

spo site classic set --url https://contoso/sites/site --noScriptSite false

Getting folder property bags

props, err := sp.Web().GetFolder("MyLibrary").Props().Get()
if err != nil {
	log.Fatal(err)
}

for key, val := range props.Data() {
	fmt.Printf("%s: %s\n", key, val)
}

Setting many properties values

err := sp.Web().GetFolder("MyLibrary").Props().
	SetProps(map[string]string{
		"prop01": "value 01",
		"prop02": "value 02",
	})

if err != nil {
	log.Fatal(err)
}

Summary

Property bags are a robust way of storing custom settings and state which requires no additional artifacts. When structuring and consuming correctly they can be a great addition to the application logic.

SharePoint is ECM (Enterprise Content Management) system and it's common to expect files being uploaded, downloaded, migrated, processes, and managed in a variety ways.

Gosip provides an easy way of dealing with SharePoint document listaries, files and folders.

Getting library object

Document library in SharePoint is almost the same as a List, but with intention of being a container for files.

// The recommended way of getting lists is by using their relative URIs
// can be a short form without full web relative URL prefix
list := sp.Web().GetList("MyLibrary")

// other common but less recommended way of getting a list is
// list := sp.Web().Lists().GetByTitle("My Library")

Getting folder object

foler := sp.Web().GetFolder("MyLibrary/Folder01")

Getting file object

file := sp.Web().GetFile("MyLibrary/Folder01/File01.txt")

Adding new folder

// folderResp is a byte array read from response body with extra methods
folderResp, err := foler.Folders().Add("New Folder Name")
if err != nil {
	log.Fatal(err)
}

fmt.Printf("New folder URL: %s\n", folderResp.Data().ServerRelativeURL)

Deleting folders

folderRelativeURL := "MyLibrary/Folder01/New Folder Name"
if _, err := sp.Web().GetFolder(folderRelativeURL).Delete(); err != nil {
	log.Fatal(err)
}

Adding/uploading a file

// fileAddResp is a byte array read from response body with extra methods
fileAddResp, err := foler.Files().
	Add("My File.txt", []byte("File content"), true)

if err != nil {
	log.Fatal(err)
}

fmt.Printf("New file URL: %s\n", fileAddResp.Data().ServerRelativeURL)

Obviously, file content can be a result of reading a file from disk, e.g.:

content, err := ioutil.ReadFile("/path/to/file.txt")
if err != nil {
	log.Fatal(err)
}

fileAddResp, err := foler.Files().Add("My File.txt", content, true)
if err != nil {
	log.Fatal(err)
}

For the large files it's better using AddChunked API, hovewer, it was not available in SharePoint 2013.

file, err := os.Open("/path/to/large/file.zip")
if err != nil {
	log.Fatalf("unable to read a file: %v\n", err)
}
defer file.Close()

fileAddResp, err := foler.Files().AddChunked("My File.zip", file, nil)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("New file URL: %s\n", fileAddResp.Data().ServerRelativeURL)

More details.

Downloading files

fileRelativeURL := "MyLibrary/Folder01/File01.txt"
data, err := sp.Web().GetFile(fileRelativeURL).Download()
if err != nil {
	log.Fatal(err)
}

file, err := os.Create("/path/toLocal/file.txt")
if err != nil {
	log.Fatalf("unable to create a file: %v\n", err)
}
defer file.Close()

_, err = file.Write(data)
if err != nil {
	log.Fatalf("unable to write to file: %v\n", err)
}

file.Sync()

For a large files it's better getting file reader through:

fileRelativeURL := "MyLibrary/Folder01/File01.txt"
fileReader, err := sp.Web().GetFile(fileRelativeURL).GetReader()
if err != nil {
	log.Fatal(err)
}
defer fileReader.Close()

file, err := os.Create("/path/toLocal/file.txt")
if err != nil {
	log.Fatalf("unable to create a file: %v\n", err)
}
defer file.Close()

if _, err := io.Copy(file, fileReader); err != nil {
	log.Fatalf("unable to save a file: %v\n", err)
}

Summary

Using Gosip you can concentrate on business logic and Go language aspects while processing documents actions in SharePoint seemlessly.

With use of IntelliSense and Fluent syntax other supported actions can be consumed based on a specific requirement.

Gosip client respects native Go Context, you can pass a context on a low level or to a Fluent API to control requests's deadlines, cancellation signals, and other request-scoped values across API boundaries and between processes.

Low level client

On the low level dealing with the contexts is identical to native approach:

client := &gosip.SPClient{AuthCnfg: auth}

var req *http.Request
// Initiate API request
// ...

req = req.WithContext(context.Background()) // <- pass a context

resp, err := client.Execute(req)
if err != nil {
  fmt.Printf("Unable to request api: %v", err)
  return
}

HTTP Client

While using HTTPClient, context is defined together with request config.

spClient := api.NewHTTPClient(&gosip.SPClient{AuthCnfg: auth})

endpoint := auth.GetSiteURL() + "/_api/web?$select=Title"

reqConf := &api.RequestConfig{
  Context: context.Background(), // <- pass a context
}

data, err := spClient.Get(endpoint, reqConf)
if err != nil {
  log.Fatalf("%v\n", err)
}

// spClient.Post(endpoint, body, reqConf) // generic POST

// generic DELETE helper crafts "X-Http-Method"="DELETE" header
// spClient.Delete(endpoint, reqConf)

// generic UPDATE helper crafts "X-Http-Method"="MERGE" header
// spClient.Update(endpoint, body, reqConf)

// CSOM helper (client.svc/ProcessQuery)
// spClient.ProcessQuery(endpoint, body, reqConf)

Fluent API

With Fluent API, context is managed in the same way as in the previous example with the only difference how it is chained to fluent syntax.

config := &api.RequestConfig{
  Context: context.Background(), // <- pass a context
}

sp := api.NewSP(client).Conf(config)

data, err := sp.Web().Lists().Select("Id,Title").Get()
if err != nil {
	log.Fatalln(err)
}

Conf method can be used almost on any hierarchy level. It's inherited with capability to redefine.

Permissions management is an important part of business logic which can be met in worker processes and workflows.

Let's explore how Gosip Fluent API wraps up SharePoint securable object and permissions.

First of all, by securable objects mean any SharePoint artifacts which can be configured with unique permissions: webs, lists, libraries, items, etc.

Fluent API scopes role operations under .Roles() method.

Checking is an object has unique permissions

item := list.Items().GetByID(1)
hasUniqueAssignments, err := item.Roles().HasUniqueAssignments()
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Has unique permissions: %t\n", hasUniqueAssignments)

Breaking role inheritance

To assign unique permissions to an object its role inheritence must be broken.

item := list.Items().GetByID(1)
if err := item.Roles().BreakInheritance(true, false); err != nil {
	log.Fatal(err)
}

// where the first argument stands for `copyRoleAssigments`
//   - if true the permissions are copied from the current parent scope
// second argument is `clearSubScopes`
//  - true to make all child securable objects inherit role assignments 
//  from the current object

Be aware that abusing breaking role inheritence and having too many unique permissions can have a detrimental effect on SharePoint performance. The architecture of a solution which involves many unique permissions has too be planned carefully. See some valuable recommendations.

Role definitions

Before assigning permissions it's important to know how to get role definitions.

A role definition is a collection of rights bound to a specific object. Role definitions (for example, Full Control, Read, Contribute, Design, or Limited Access) are scoped to the Web site and mean the same thing everywhere within the Web site, but their meanings can differ between sites within the same site collection. Role definitions can also be inherited from the parent Web site, just as permissions can be inherited.

Definitions are csoped as Web level in .RoleDefinitions() quariable collection.

def, err := sp.Web().RoleDefinitions().GetByType(api.RoleTypeKinds.Contributor)
if err != nil {
	log.Fatal(err)
}

Definitions can be gotten by Definition ID, Name or Type. We recommend using OOTB definitions and getting them by types. There is an enumerator-like helper for default types api.RoleTypeKinds.

Getting principals

Principal is site user or group, their IDs are used in roles assignments. Principal ID can be received by requesting UIL (User Information List), getting site user/group by name/email, etc.

roup, err := sp.Web().SiteGroups().GetByName("Process Users").Get()
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Principal ID: %d\n", group.Data().ID)

// or

user, err := sp.Web().SiteUsers().GetByEmail("jonh.doe@contoso.com").Get()
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Principal ID: %d\n", user.Data().ID)

It's in preference to operate on groups level and grant permissions to a specific users as little as possible.

Adding role assignments

The role assignment is the relationship among the role definition, the users and groups, and the scope (for example, one user may be a reader on list 1, while another user is a reader on list 2). The relationship expressed through the role assignment is the key to making SharePoint security management role-based.

At last, now we are ready for roles assigment. 😜 Who told permissions is simple?

if err := item.Roles().AddAssigment(group.Data().ID, def.ID); err != nil {
	log.Fatal(err)
}

Removing role assignments

Removing role assignments is just the same as adding but in opposite.

if err := item.Roles().RemoveAssigment(group.Data().ID, def.ID); err != nil {
	log.Fatal(err)
}

Reseting roles inheritance

To reset permissions inheritance:

item := list.Items().GetByID(1)
if err := item.Roles().ResetInheritance(); err != nil {
	log.Fatal(err)
}

After reseting object roles assigments its permissions are again inherited from the parent object.

Getting collection objects' permissions

When dealing with OData collection of objects with unique permissions OData items' role assigment can be requested using RoleAssignments, also HasUniqueRoleAssignments can be used in moderation.

Please be aware that HasUniqueRoleAssignments is a heavy property that creates workload on a SharePoint server and better be used at minimal.

Try not abusing it by potential requests to large lists getting a bunch of items.

// Getting all lists with role assigments details
res, err := sp.Web().Lists().
  Select(`
    Title,
    RoleAssignments/Member/*,
    RoleAssignments/RoleDefinitionBindings/*
  `).
  Expand(`
    RoleAssignments/Member,
    RoleAssignments/RoleDefinitionBindings
  `).
  Get()

See more.

RoleAssigments if any applied contains an array of objects.

var lists []*struct {
  Title           string
  RoleAssignments []*api.RoleAssigment
}

// .Normalized() method aligns responses between different OData modes
if err := json.Unmarshal(res.Normalized(), &lists); err != nil {
  log.Fatalf("unable to parse the response: %v", err)
}

Assigments is a Member and binded RoleDefinitions:

// RoleAssigment role asigments model
type RoleAssigment struct {
	Member *struct {
		LoginName     string
		PrincipalType int
	}
	RoleDefinitionBindings []*RoleDefInfo
}

Base permissions

BasePermissions is permissions representation with Low and High pair. Don't panic if API returns only BasePermissions ({ "High": "2147483647", "Low": "4294705151" }), using HasPermissions helper it's simple to check if it includes required permissions kind:

// User effective base permissions
effectiveBasePermissions := api.BasePermissions{
	High: 432,
	Low:  1011030767,
}

hasPerm := api.HasPermissions(
  effectiveBasePermissions,
  api.PermissionKind.EditListItems)

Summary

Now we know how to treat SharePoint permissions using Gosip and Fluent API. Before wrapping up, we want to stress on importance of careful planning of permissions model, as less unique permissions is better.

For large documents it's better using files Chunk API which allows splitting upload into separate REST API requests reducing memory consumption and providing a more manageable upload mechanism.

Files Chunk API appeared in SharePoint 2016 and should not be expected to work in the previous version.

Basic chunk upload example

file, err := os.Open("/path/to/large/file.zip")
if err != nil {
	log.Fatalf("unable to read a file: %v\n", err)
}
defer file.Close()

fileAddResp, err := foler.Files().AddChunked("My File.zip", file, nil)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("New file URL: %s\n", fileAddResp.Data().ServerRelativeURL)

Upload flow

Fluent API abstracts some aspects of the chunked upload. Internally, /StartUpload, /ContinueUpload, /CancelUpload, and /FinishUpload endpoints are used. However, these are not exposed for the simplicity. Nonetheless, there is a way for canceling upload with the help of api.AddChunkedOptions.

Add chunked options

There are a few options which are optional. The third method parameter is a configuration structure.

type AddChunkedOptions struct {
	Overwrite bool // should overwrite existing file
	// on progress callback, execute custom logic on each chunk
	// if the Progress is used it should return "true" to continue upload
	// otherwise upload is canceled
	Progress  func(data *FileUploadProgressData) bool
	ChunkSize int // chunk size in bytes
}

Defaults are: Overwrite is true and ChunkSize equals to 10485760 bytes.

Progress callback allows not only trigger a progress logic, for example upload percentage update, but also to cancel an upload. To cancel an upload the progress callback should return false.

filePath := "/path/to/large/file.zip"

file, err := os.Open(filePath)
if err != nil {
	log.Fatalf("unable to read a file: %v\n", err)
}
defer file.Close()

info, _ := os.Stat(filePath)

options := &api.AddChunkedOptions{
	Overwrite: false,
	ChunkSize: 5 * 1024 * 1024,
	Progress: func(data *api.FileUploadProgressData) bool {
		fmt.Printf(
			"Block %d, sent %d%%\n",
			data.BlockNumber,
			100*data.FileOffset/fi.Size(),
		)
		return true // to cancel an upload on some external condition, return "false"
	},
}

if _, err := foler.Files().AddChunked("My File.zip", file, options); err != nil {
	log.Fatal(err)
}

REST API uses OData modes for controlling response verbosity.

By defining different OData modes (Verbose, Minimalmetadata, Nometadata) within the Accept headers SharePoint REST API returns not only data of different details but also data in different forms payload shape-terms. Which can lead to runtime errors.

When dealing with different versions of SharePoint there are few gotchas to remember. In old SharePoint 2013, only Verbose mode was allowed by default. This can be amended by installing WCF OData extensions and enabling JSON Light support, however, in our practice, a rare farm admin considering such an update. So it's better stick with a Verbose when it comes to SharePoint 2013.

With SharePoint 2016 and newer, and, obviously, SharePoint Online, we'd recommend Minimalmetadata as a default. So the payloads could be a bit more smaller in size and effective overall.

Nometadata mode could be tricky, from one hand it's close to Minimalmetadata yet doesn't content some vital information such as entity identities and paged collections helpers. We prefer Minimalmetadata over Nometadata in general.

Gosip provides some presets for headers which could be handy together with Conf method.

var client *gosip.SPClient
// ...
sp := api.NewSP(client).Conf(api.HeadersPresets.Minimalmetadata)
// api.HeadersPresets.Verbose
// api.HeadersPresets.Nometadata

Along with OData modes, these presets define language header "Accept-Language": "en-US,en;q=0.9" which forces English messages in responses if English is installed on a site. Which is handy for dev and debugging purposes as sometimes a local non-latin language can be escaped to an unreadable form making it uncomfortable detecting what was wrong in logs.

You can activate and deactivate features on sites and webs using REST API. However, there are a few nuances to know knowing which makes it simple to manage the features.

Unfortunately, there is no way of getting features list with names and descriptions programmatically. When getting features list on the web or site you receive a list of activated features definition IDs and nothing else.

Getting activated features

res, err := sp.Web().Features().Get()
if err != nil {
	log.Fatal(err)
}

for _, f := range res {
	fmt.Printf("%+v\n", f)
}

// &{DefinitionID:b77b6484-364e-4356-8c72-1bb55b81c6b3}
// &{DefinitionID:a7a2793e-67cd-4dc1-9fd0-43f61581207a}
// &{DefinitionID:d5a4ed08-27b9-4142-9804-45dec6fda126}
// &{DefinitionID:780ac353-eaf8-4ac2-8c47-536d93c03fd6}
// &{DefinitionID:8c6f9096-388d-4eed-96ff-698b3ec46fc4}
// ...

Luckily, nowadays one almost always deal only with OOTB features, if not probably something is terribly wrong on a project. 😝

Feature activating

mds := "87294c72-f260-42f3-a41b-981a2ffce37a"
if err := sp.Web().Features().Add(mds, true); err != nil {
	log.Fatal(err)
}

The first argument stands for Feature Definition ID, the second one is force mode.

Feature deactivating

mds := "87294c72-f260-42f3-a41b-981a2ffce37a"
if err := sp.Web().Features().Remove(mds, true); err != nil {
	log.Fatal(err)
}

Arguments are identical.

When a feature was not activated before removing it an error message is expected ("Feature is not activated at this scope").

Site level actions are absolutely identical, the only difference is sp.Site() entry point.

When it comes to synchronization or delta processing Change API is for the rescue. There is no need in any sort of logic that used modified date comparison to detect what has been changed since the last processing time. There is no need to wait until the search crawl is done to check changes through web or site collection. And not only these benefits but also tracking permissions changes restores from recycle bin, system updates and much more.

Changes API is what is used together with Webhooks (in SPO), by having a context (scope) and change token(s) you can easily retrieve what has been changed and run the custom logic based on the nature of changes.

Changes can be requested from within different scopes: lists, webs, sites, etc. Change API operates change tokens to establish start and end anchors.

Change tokens received from a specific entity type (e.g. Site) can't be used with another entity type (e.g. List) while sending change query.

Getting current change token

siteChangeToken, err := sp.Site().Changes().GetCurrentToken()
if err != nil {
  log.Fatal(err)
}

fmt.Printf("Site change token: %s\n", siteChangeToken)

webChangeToken, err := sp.Web().Changes().GetCurrentToken()
if err != nil {
  log.Fatal(err)
}

fmt.Printf("Web change token: %s\n", webChangeToken)

list := sp.Web().GetList("Lists/MyList")
listChangeToken, err := list.Changes().GetCurrentToken()
if err != nil {
  log.Fatal(err)
}

fmt.Printf("List change token: %s\n", listChangeToken)

// Site change token:
//   1;1;afbc6f5a-65b3-4b64-9f40-885d8d772c8c;637138201430100000;64696075
// Web change token:
//   1;2;7e084f68-c401-4910-b75e-4347c7848965;637138201430100000;64696075
// List change token:
//   1;3;3b542696-5863-4ff7-bb90-8224e9e8adb9;637138201430100000;64696075

Getting changes

Knowing what root entities' token(s) you have the specific changes query can be crafted and sent, e.g.:

list := sp.Web().GetList("Lists/MyList")
listChangeToken, _ := list.Changes().GetCurrentToken()
list.Items().Add([]byte(`{"Title":"New item"}`))
// error handling is omitted -- adding a dummy item to receive changes result

changes, err := list.Changes().GetChanges(&api.ChangeQuery{
  ChangeTokenStart: listChangeToken,
  List:             true,
  Item:             true,
  Add:              true,
})
if err != nil {
  log.Fatal(err)
}

for _, change := range changes.Data() {
  fmt.Printf("%+v\n", change)
}

Change results are strongly typed with Gosip's API, the changes variable in the sample is an array of api.ChangeInfo struct pointers.

Change into struct contains the following properties:

type ChangeInfo struct {
  ChangeToken       *StringValue
  ChangeType        int
  Editor            string
  EditorEmailHint   string
  ItemID            int
  ListID            string
  ServerRelativeURL string
  SharedByUser      string
  SharedWithUsers   string
  SiteID            string
  Time              time.Time
  UniqueID          string
  WebID             string
}

ChangeType is an important piece of information, it's an enumerator describing a nature of a change. See more.

In a synchronization scenario or a Webhook after getting information which items are changed the corresponding request(s) should be sent for getting specifics.

Change query

When getting the changes the API requires some clarifications about what exactly should be retrieved. This is defined within a change query which is implemented api.ChangeQuery struct.

Pagination

Sometimes it can be lots of changes since a provided token. Gosip implements pagination helper using "jumping" between different change tokens under the hood. When last change item's token is used as a start token to get the "next page". This approach is used in GetNextPage:

changesFirstPage, _ := list.Changes().Top(100).GetChanges(&ChangeQuery{
	ChangeTokenStart: token,
	List:             true,
	Item:             true,
	Add:              true,
})

changesSecondPage, _ := changesFirstPage.GetNextPage()

Summary

Change API is a powerful mechanism and a robust way for processing delta changes that can and should be used in advanced and optimised synchronizations and business processes withing external workers.

Working with UPS API is simple. UPS API is not something sophisticated as Search for example and it has just a few methods which can be in demand in a server-side operation.

Don't get me wrong, there are not-covered social features from within user profiles API in Gosip but we rarely have seen demand and scenarios of their usage in SharePoint solutions. Anyways, if some additional feature coverage is required for you please let us know by posting an issue.

Getting profiles

The most in-demand, I would say, feature when it comes to user profiles is getting all profiles. However, UPS API allows only dealing with a single profile, you can't request all of them.

Luckily, this is possible and recommended achieving with the search.

res, err := sp.Search().PostQuery(&api.SearchQuery{
	QueryText: "*",
	SourceID:  "b09a7990-05ea-4af9-81ef-edfab16c4e31",
})

if err != nil {
	log.Fatal(err)
}

fmt.Printf("%+v\n", res.Results())

See a bit more advanced sample.

Getting profile properties

user, err := sp.Web().SiteUsers().
	GetByEmail("jane.doe@contoso.onmicrosoft.com").Get()

if err != nil {
	log.Fatal(err)
}

props, err := sp.Profiles().GetPropertiesFor(user.Data().LoginName)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("%+v\n", props.Data())

Gosip strongly types properties response using .Data() helper, here is the resulted struct:

type ProfilePropsInto struct {
	AccountName           string
	DirectReports         []string
	DisplayName           string
	Email                 string
	ExtendedManagers      []string
	ExtendedReports       []string
	Peers                 []string
	IsFollowed            bool
	PersonalSiteHostURL   string
	PersonalURL           string
	PictureURL            string
	Title                 string
	UserURL               string
	UserProfileProperties []*TypedKeyValue
}

Getting single property

Sometimes you have to be as effective as possible and trim down responses to a minimum. Let's say you only need a single property.

rop, err := sp.Profiles().
	GetUserProfilePropertyFor(user.Data().LoginName, "AccountName")

if err != nil {
	log.Fatal(err)
}

fmt.Printf("%s\n", prop)

Seriously, I don't know the reason for existing of getting a single property, but not getting specific properties or multiple profiles or updating multiple properties. There are no excuses yet this part of SharePoint API is clunky and really old.

Setting user profile property values

There are two methods for setting user profile property value. Yeah, you heard me the right property in a time.

Set single value profile property

if err := sp.Profiles().SetSingleValueProfileProperty(
	user.Data().LoginName,
	"AboutMe",
	"Updated from Gosip",
); err != nil {
	log.Fatal(err)
}

Set multi valued profile property

tags := []string{"#ci", "#demo", "#test"}
if err := sp.Profiles().SetMultiValuedProfileProperty(
	user.Data().LoginName,
	"SPS-HashTags",
	tags,
); err != nil {
	log.Fatal(err)
}

Summary

User profiles API is limited due to its legacy nature. It is what it is. However, many SharePoint solutions, especially intranet portals and workflow processes can be heavily based on UPS. Go worker can be handy for custom synchronization scenarios and also in external workflow workers with UPS as a source for settings for detecting user dynamic roles.

Site groups and users can be requested via Web's .SiteGroups() and .SiteUsers() queriable collections correspondingly.

Getting users sample

users, err := sp.Web().SiteUsers().
	Select("Email,Id").
	Filter("Email ne ''").
	OrderBy("Id", true).
	Get()

if err != nil {
	log.Fatal(err)
}

for _, user := range users.Data() {
	fmt.Printf("%d: %s\n", user.Data().ID, user.Data().Email)
}

Getting groups sample

groups, err := sp.Web().SiteGroups().Get()
if err != nil {
	log.Fatal(err)
}

for _, group := range groups.Data() {
	fmt.Printf("%d: %s\n", user.Data().ID)
}

Ensuring a user

You can't add new user via SharePoint API, but a user who exists in AD/AAD can be added to a site by ensuring him/her by a logon name.

user, err := sp.Web().EnsureUser("jane.doe@contoso.onmicrosoft.com")
if err != nil {
	log.Fatal(err)
}

fmt.Printf("User: %+v\n", user)

Associated groups

We love the "power of defaults". Each web by default has three predefined groups: Owners, Members and Visitors. But their IDs and names are different from web to web. Luckily there is a helper for getting associated groups.

// .Members() and .Owners() correspondingly
group, err := sp.Web().AssociatedGroups().Visitors().Get()
if err != nil {
	log.Fatal(err)
}

fmt.Printf(
  "Visitors group ID %d, Title \"%s\"\n",
  group.Data().ID,
  group.Data().Title,
)

Creating groups

group, err := sp.Web().SiteGroups().Add("My group", nil)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("New group ID: %d\n", group.Data().ID)

Deleting groups

// or .RemoveByID(groupID)
if err := sp.Web().SiteGroups().RemoveByLoginName("My group"); err != nil {
	log.Fatal(err)
}

Adding user to a group

user, err := sp.Web().EnsureUser("jane.doe@contoso.onmicrosoft.com")
if err != nil {
	log.Fatal(err)
}

fmt.Printf("User login: %s\n", user.LoginName)
// User login: i:0#.f|membership|jane.doe@contoso.onmicrosoft.com

visitorGroup := sp.Web().AssociatedGroups().Visitors()
if err := visitorGroup.AddUser(user.LoginName); err != nil {
	log.Fatal(err)
}

In group's .AddUser method the argument should be full and valid login name including security provider membership prefix. For instance, while you can ensure Jane using jane.doe@contoso.onmicrosoft.com the same as an .AddUser method will fail as Jane's login is actually different i:0#.f|membership|jane.doe@contoso.onmicrosoft.com.

If you already know UserID but not sure about LoginName .AddUserByID helper is at the disposal.

Removing users from a group

Similarly as with adding users:

user, err := sp.Web().EnsureUser("jane.doe@contoso.onmicrosoft.com")
if err != nil {
	log.Fatal(err)
}

memberGroup := sp.Web().AssociatedGroups().Members()
if err := memberGroup.RemoveUser(user.LoginName); err != nil {
	log.Fatal(err)
}

// or

if err := memberGroup.RemoveUserByID(user.ID); err != nil {
	log.Fatal(err)
}

Managing group owner

memberGroup := sp.Web().AssociatedGroups().Members()
if err := memberGroup.SetAsOwner(user.ID); err != nil {
	log.Fatal(err)
}

Getting group's users

users, err := sp.Web().AssociatedGroups().Visitors().
	Users().Select("Id,Title").Get()

if err != nil {
	log.Fatal(err)
}

for _, user := range users.Data() {
	fmt.Printf("%d: %s\n", user.Data().ID, user.Data().Title)
}

Summary

That's it, most of the common actions with groups and users are covered.

You'd probably will be interested with the connected topics:

Intro

First of all, thank you for considering contributing to the project! We really appreciate any activity around it. There are no small contributions and any investment can't be underestimated. You can contribute with feedback, finding and posting an issue, docs suggestion, code commit, or a star ⭐️. All of these encourage us supporting this and other our Open Source projects on a high level.

Contributing to the project follows a majority of GitHub and Open Source communities' principles.

Code contributing guidance

• Target your pull requests to the dev branch.

• Add/update any docs articles related to your changes embeded to the code or as a separate notes to the PR, which we'd love to publish on docs site.

• • If you are fixing a bug, include a test that would have caught the bug you are fixing.

• Keep your PRs as simple as possible and describe the changes to help the reviewer understand your work.

When it comes to a code base which should support multiple platform versions, which APIs obviously changes with time and not aligned together, it can be a challenge maintaining-wise.

Luckily, authentication methods are isolated and pretty static and HTTP client is generic for any SharePoint API consumption, no-matter REST, CSOM, or legacy SOAP are intended to be used.

In enterprise content management (ECM) scenarios, records are an important element within a document lifecycle and a company's compliance policies. Not only documents but actually items.

When declared as a record a document or item can't be later changed or moved without undeclaring. This guarantees integrity.

REST API in terms of managing classic in-place records is limited, you can only retrieve record status from item metadata. Item's OData__vti_ItemDeclaredRecord property tells was an item declared as a record and when. The field is a date-time value, empty means item is not a record.

We would not be we if didn't add an extension with allows Gosip to declare, undeclare and declare with a declaration date. To workaround REST's limitation, these functionality is achieved by corresponding CSOM calls. What's great is that Gosip abstracts for you such a nuance providing methods which under-the-hood details are interesting but not necessarily should be known by a library consumer.

Is item declared as record

Is document declared as a record

Declaration as records is more common for documents (files), yet it's actually a file's item that keeps record status. And therefore holds API actions.

Declare as a record

Undeclare as a record

Declare with declaration date

Summary

Cpass is simplified secured password two-ways encryption sub package port for Gosip.

By default, Cpass uses Machine ID as an encryption key so a secret hash can only be decrypted on a machine where it was generated.

Cpass's approach is appropriate in local development scenarios. The main goal is "not to show raw secret while presenting a desktop" or "not to commit raw secret by an incident to code source".

Installation

go get github.com/koltyakov/gosip/cpass

Convertor

package main

import (
	"flag"
	"fmt"

	"github.com/koltyakov/gosip/cpass"
)

func main() {

	var rawSecret string

	flag.StringVar(&rawSecret, "secret", "", "Raw secret string")
	flag.Parse()

	crypt := cpass.Cpass("")

	secret, _ := crypt.Encode(rawSecret)
	fmt.Println(secret)

}

Encrypt secrets

go run ./ -secret "MyP@s$word"
#> -lywbAGD4iPYdJXDxLAQoMUbfBXBIQR2UZYl

When use result token/hash as a secret in private.json file(s).

From sandbox

Another option would be installing cpass from sandbox:

go install github.com/koltyakov/gosip-sandbox/samples/cpass

And using cpass as a CLI, with no parameters the secret can be provided in a masked form without keeping it in console history:

$ cpass
Password to encode: ********
poXx8zaJM6gLazPCtv4rMVLoTuzX_1BvYJlMAQqK

There is nothing simpler than sending email notifications using SharePoint REST and Gosip Fluent API. However, when building workflow workers or custom subscription services email functionality is vital. And what can be better than embedded OOTB functionality? No extra knowledge of SMTP server and mail credentials, just a usual API call to _api/SP.Utilities.Utility.SendEmail utility endpoint.

There are some limitations to this method:

• recipients only from the site

• impossibility to change the sender

• no attachments

but as embedded solution it's OK. And probably sort of workflow notification service should stay in such margins anyways.

Sending email notification

user, err := sp.Web().SiteUsers().
	GetByEmail("jane.doe@contoso.onmicrosoft.com").Get()

if err != nil {
	log.Fatal(err)
}

if err := sp.Utility().SendEmail(&api.EmailProps{
	Subject: "Say hi to Gosip!",
	Body:    "Text or HTML body here...",
	To:      []string{user.Data().Email},
}); err != nil {
	log.Fatal(err)
}

Send email utility is not sophisticated but it works and enough for an embedded functionality.

In a clone or fork.

Authentication testing

Create auth credentials store files in ./config folder for corresponding strategies:

• private.onprem-adfs.json

• private.onprem-fba.json

• private.onprem-ntlm.json

• private.onprem-tmg.json

• private.onprem-wap-adfs.json

• private.onprem-wap.json

• private.spo-addin.json

• private.spo-user.json

• private.spo-adfs.json

See samples.

go test ./... -v -race -count=1

Not provided auth configs and therefore strategies are ignored and not skipped in tests.

API integration tests

Create auth credentials store files in ./config/integration folder for corresponding environments:

• private.spo.json

• private.2013.json // 2013 has its nuances with OData mod and not supported methods

SPAUTH_ENVCODE=spo go test ./api/... -v -race -count=1
SPAUTH_ENVCODE=2013 go test ./api/... -v -race -count=1

API integration tests are mostly targeted to SharePoint Online and not regularly processed on the legacy versions of the platform so you can face some test exceptions which still should be escaped with t.Skip and envCode != "spo" condition:

if envCode != "spo" {
  t.Skip("is not supported with old SharePoint versions")
}

Environment variables

• SPAUTH_ENVCODE=code environment variable switches target environments. spo is a default one.

• SPAPI_HEAVY_TESTS=true turns on "heavy" methods, e.g. web creation.

Run manual tests

Modify cmd/test/main.go to include required scenarios and run:

go run ./cmd/test

Optionally, you can provide a strategy to use with a corresponding flag:

go run ./cmd/test -strategy adfs

Run CI tests

Configure environment variables:

• SPAUTH_SITEURL

• SPAUTH_CLIENTID

• SPAUTH_CLIENTSECRET

• SPAUTH_USERNAME

• SPAUTH_PASSWORD

go test ./... -v -race -count=1
SPAUTH_ENVCODE=spo SPAPI_HEAVY_TESTS=true go test ./api/... -v -race -count=1

Test coverage

Check Codecov.

We are targeted to keep code coverage higher than 80% for API and Auth methods altogether.

Search and search API is a huge and complex topic. In this article we're going mostly cover some basics in combination of Gosip plus REST API.

Basic search call

While executing search call there is no difference what web/site is used as a context, search API returns data due to search query object.

The simplest request looks this way:

// `res`ult here is a byte array with some helper methods as .Results()
res, err := sp.Search().PostQuery(&api.SearchQuery{
    QueryText: "*",
    RowLimit:  5,
})

if err != nil {
    log.Fatal(err)
}

fmt.Printf("%+v\n", res.Results())

Search query

Search query struct (api.SearchQuery) contains some, let's say, reasonable amount of options 🙈.