# Introduction ⚡️ SharePoint SDK for Go (Golang) ![Build Status](https://koltyakov.visualstudio.com/SPNode/_apis/build/status/gosip?branchName=master) [![Go Report Card](https://goreportcard.com/badge/github.com/koltyakov/gosip)](https://goreportcard.com/report/github.com/koltyakov/gosip) [![codecov](https://codecov.io/gh/koltyakov/gosip/branch/master/graph/badge.svg)](https://codecov.io/gh/koltyakov/gosip) [![License](https://img.shields.io/github/license/koltyakov/gosip.svg)](https://github.com/koltyakov/gosip/blob/master/LICENSE) [![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go) ![](/files/-Lxep0QlsK2R-f-E51AH) ## Main features * Unattended authentication using different strategies. * [Fluent API](/api/fluent) syntax for SharePoint object model. * Simplified API consumption (REST, CSOM, SOAP, etc.). * SharePoint-aware embedded features (retries, header presets, error handling). ### Supported SharePoint versions * SharePoint Online (SPO). * On-Premises (2019/2016/2013). ### Supported auth strategies * SharePoint Online: * [Azure Certificate (App Only)](/auth/strategies/azure-certificate-auth) * [Azure Username/Password](/auth/strategies/azure-creds-auth) * [SAML](/auth/strategies/saml) based with user credentials * [Add-In only](/auth/strategies/addin) permissions * ADFS user credentials (automatically detects in [SAML](/auth/strategies/saml) strategy) * [Azure Device flow](/auth/strategies/azure-device-flow) * [On-Demand authentication](/auth/strategies/on-demand) * SharePoint On-Premises 2019/2016/2013: * [User credentials](/auth/strategies/ntlm) (NTLM) * [ADFS user credentials](/auth/strategies/adfs) (ADFS, WAP -> Basic/NTLM, WAP -> ADFS) * Behind a reverse proxy ([Forefront TMG](/auth/strategies/tmg), [WAP -> Basic/NTLM](/auth/strategies/adfs), [WAP -> ADFS](/auth/strategies/adfs)) * [Form-based authentication](/auth/strategies/fba) (FBA) * [On-Demand authentication](/auth/strategies/on-demand) ## Installation ```bash go get github.com/koltyakov/gosip ``` ## Usage insights ### Understand SharePoint environment type and authentication strategy Let's assume it's SharePoint Online and Add-In Only permissions. Then `strategy "github.com/koltyakov/gosip/auth/addin"` sub package should be used. ```go package main import ( "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/addin" ) ``` ### Initiate authentication object ```go auth := &strategy.AuthCnfg{ SiteURL: os.Getenv("SPAUTH_SITEURL"), ClientID: os.Getenv("SPAUTH_CLIENTID"), ClientSecret: os.Getenv("SPAUTH_CLIENTSECRET"), } ``` AuthCnfg's from different strategies contains different options relevant for a specified auth type. The authentication options can be provided explicitly or can be read from a configuration file (see [more](/auth/overview#authentication-strategies)). ```go configPath := "./config/private.json" auth := &strategy.AuthCnfg{} err := auth.ReadConfig(configPath) if err != nil { fmt.Printf("Unable to get config: %v\n", err) return } ``` ### Bind auth client with Fluent API ```go client := &gosip.SPClient{AuthCnfg: auth} sp := api.NewSP(client) res, err := sp.Web().Select("Title").Get() if err != nil { fmt.Println(err) } fmt.Printf("%s\n", res.Data().Title) ``` ## Usage samples ### Fluent API client Provides a simple way of constructing API endpoint calls with IntelliSense and chainable syntax. ```go package main import ( "encoding/json" "fmt" "log" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/addin" ) func main() { // Getting auth params and client client, err := getAuthClient() if err != nil { log.Fatalln(err) } // Binding SharePoint API sp := api.NewSP(client) // Custom headers headers := map[string]string{ "Accept": "application/json;odata=minimalmetadata", "Accept-Language": "de-DE,de;q=0.9", } config := &api.RequestConfig{Headers: headers} // Chainable request sample data, err := sp.Conf(config).Web().Lists().Select("Id,Title").Get() if err != nil { log.Fatalln(err) } // Response object unmarshalling // (struct depends on OData mode and API method) res := &struct { Value []struct { ID string `json:"Id"` Title string `json:"Title"` } `json:"value"` }{} if err := json.Unmarshal(data, &res); err != nil { log.Fatalf("unable to parse the response: %v", err) } for _, list := range res.Value { fmt.Printf("%+v\n", list) } } func getAuthClient() (*gosip.SPClient, error) { configPath := "./config/private.spo-addin.json" // <- file with creds auth := &strategy.AuthCnfg{} if err := auth.ReadConfig(configPath); err != nil { return nil, fmt.Errorf("unable to get config: %v", err) } return &gosip.SPClient{AuthCnfg: auth}, nil } ``` ### Generic HTTP-client helper Provides generic GET/POST helpers for REST operations, reducing amount of `http.NewRequest` scaffolded code, can be used for custom or not covered with a Fluent API endpoints. ```go package main import ( "fmt" "log" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/ntlm" ) func main() { configPath := "./config/private.ntlm.json" auth := &strategy.AuthCnfg{} if err := auth.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v\n", err) } sp := api.NewHTTPClient(&gosip.SPClient{AuthCnfg: auth}) endpoint := auth.GetSiteURL() + "/_api/web?$select=Title" data, err := sp.Get(endpoint, nil) if err != nil { log.Fatalf("%v\n", err) } // sp.Post(endpoint, []byte(body), nil) // generic POST // generic DELETE helper crafts "X-Http-Method"="DELETE" header // sp.Delete(endpoint, nil) // generic UPDATE helper crafts "X-Http-Method"="MERGE" header // sp.Update(endpoint, nil) // CSOM helper (client.svc/ProcessQuery) // sp.ProcessQuery(endpoint, []byte(body)) fmt.Printf("response: %s\n", data) } ``` ### Low-level HTTP-client usage Low-lever SharePoint-aware HTTP client from `github.com/koltyakov/gosip` package for custom or not covered with a Fluent API client endpoints with granular control for HTTP request, response, and `http.Client` parameters. Used internally but almost never required in a consumer code. ```go client := &gosip.SPClient{AuthCnfg: auth} var req *http.Request // Initiate API request // ... resp, err := client.Execute(req) if err != nil { fmt.Printf("Unable to request api: %v", err) return } ``` SPClient has `Execute` method which is a wrapper function injecting SharePoint authentication and ending up calling `http.Client`'s `Do` method. ## Reference Many auth flows have been "copied" from [node-sp-auth](https://github.com/s-kainet/node-sp-auth) library (used as a blueprint), which we intensively use in Node.js ecosystem for years. Fluent API and wrapper syntax are inspired by [PnPjs](https://github.com/pnp/pnpjs) which is also the first-class citizen on almost all our Node.js and front-end projects with SharePoint involved. ## License [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fkoltyakov%2Fgosip.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fkoltyakov%2Fgosip?ref=badge_large) # FAQ Frequently asked questions ## Why Golang? Go has many strong parts. It's damn simple, friendly for large team projects, super fast in a compilation, fast in runtime and requires little or no prerequisites. It is friendly for concurrent processes, modern hardware architectures, network applications and cloud-based solutions, DevOps tool. Go promises backward compatibility and long support period. Many say that Go is "the next enterprise big thing" and one of the default preferred languages. Lots of modern huge and awesome applications are written in Go while staying simple and reliable. Many main vendors use Go in one way or another. Go has lots of [wonderful libraries](https://github.com/avelino/awesome-go) at disposal. But little or no for SharePoint... until Gosip. So it's our spot to provide such an option. ## For whom is this for? In the first place, Gosip is for developers who use Go on their daily basis and who got to integrate their apps with SharePoint without diving too deep to learn it from the grouds up. SharePoint is complicated. We spent years solving different complex and misc problems with the platform. Not only for the end-users and customers but also for other developers and Open Source community. Recently more and more of our internal tools have been written in Go. Our opinion is that a tool is better if it is used in a variety of different scenarios and by different teams. So we decided to deliver our Go+SP experience to Open Source and improve it together with you. If you have no idea what is Go probably Gosip is not for you, but if you have no "Why Go?" question Gosip is an option to master SharePoint API with ease. ## Where to use? Anywhere where you considering an application written in Go reasonable. Use-cases can vary, let me describe just a few we use the library intensively: * Web services and APIs (dedicated web API consumed by many clients) * Microservices applications * Schedule runners, workflow workers, queue listeners * CLI based tools & scripting * Relays and gateways with elements of API communication * Development toolchains We would love to hear your unique use-case and even help you with our opinion. ## Why not C#, PowerShell or Node.js Under any circumstances, we're not even suggesting replacing anything with anything. It's just wrong positioning. Gosip and Go, in particular, is yet another available option that might or might not suit one needs. Even more, we use all the combination of technologies together following common sense, expertise, constraints, and many more factors. Sometimes something in Go helps in a PowerShell script. Sometimes a great .Net library is used together in Go worker. And don't forget we're also sort of ambassadors of [Node.js ecosystem for SharePoint](https://github.com/SharePoint-NodeJS/Awesome-SharePoint-Node.js). Together is stronger. For no reason don't search for a holly war aspect use whatever feels better for you for delivering great product or service. # Overview 🔐 SharePoint authentication strategies implemented in Gosip ### Authentication strategies Auth strategy should be selected corresponding to your SharePoint environment and its configuration. Import path `strategy "github.com/koltyakov/gosip/auth/{strategy}"`. Where `/{strategy}` stands for a strategy auth package. | `/azurecert` | ✅ | ❌ | [sample](/auth/strategies/azure-certificate-auth#json) | | ------------- | -------------- | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/azurecreds` | ✅ | ❌ | [sample](/auth/strategies/azure-creds-auth#json) | | `/azureenv` | ✅ | ❌ | [sample](/auth/strategies/azure-environment-auth) | | `/device` | ✅ | ❌ | [sample](/auth/strategies/azure-device-flow#auth-configuration-and-usage) | | `/addin` | ✅ | ❌ | [sample](/auth/strategies/addin#json) | | `/ntlm` | ❌ | ✅ | [sample](/auth/strategies/ntlm#json) | | `/adfs` | ✅ | ✅ | [spo](/auth/strategies/adfs#sharepoint-online-configuration), [on-prem](/auth/strategies/adfs#on-premises-configuration), [on-prem (wap)](/auth/strategies/adfs#on-premises-behing-wap-configuration) | | `/fba` | ❌ | ✅ | [sample](/auth/strategies/fba#json) | | `/tmg` | ❌ | ✅ | [sample](/auth/strategies/tmg#json) | | `/saml` | ❌ (deprecated) | ❌ | [sample](/auth/strategies/addin#json) | JSON and struct representations are different in terms of language notations. So credentials parameters names in `private.json` files and declared as structs initiators vary. ### Additional strategies Gosip supports [custom](/auth/custom-auth) (ad hoc) strategies. Some worthy are boiled in [the Sandbox](https://github.com/koltyakov/gosip-sandbox/tree/master/strategies) to be added later on to the main package in a case of the demand. | Strategy name | SPO | On-Prem | Credentials sample(s) | | ---------------- | --- | ------- | --------------------------------------------------------------- | | On-Demand | ✅ | ✅ | [sample](/auth/strategies/on-demand#configure-and-usage-sample) | | Alternative NTLM | ❌ | ✅ | [see more](/auth/strategies/alternative-ntlm) | ### Secrets encoding When storing credential in local `private.json` files, which can be handy in local development scenarios, we strongly recommend to encode secrets such as `password` or `clientSecret` using [cpass](/utilits/cpass). Cpass converts a secret to an encrypted representation which can only be decrypted on the same machine where it was generated. This minimize incidental leaks, i.e. with git commits. # Strategies 🔐 SharePoint authentication strategies implemented in Gosip ### Supported strategies {% tabs %} {% tab title="SharePoint Online" %} {% content-ref url="/pages/-MQc\_NXE0IfcTWInGOTz" %} [Azure Certificate Auth](/auth/strategies/azure-certificate-auth) {% endcontent-ref %} {% content-ref url="/pages/-MGYbYdtkO\_B1cakbTop" %} [Azure Env-based Auth](/auth/strategies/azure-environment-auth) {% endcontent-ref %} {% content-ref url="/pages/-MQcbyczaP1HIwpE6oj2" %} [Azure Creds Auth](/auth/strategies/azure-creds-auth) {% endcontent-ref %} {% content-ref url="/pages/-LxwUUXVydI33\_am6Ikd" %} [Azure Device Flow](/auth/strategies/azure-device-flow) {% endcontent-ref %} {% content-ref url="/pages/-LxevGKlDqci2TEIioct" %} [AddIn Only](/auth/strategies/addin) {% endcontent-ref %} {% content-ref url="/pages/-LxeviGxgiisbzKNDkOb" %} [ADFS Auth](/auth/strategies/adfs) {% endcontent-ref %} {% endtab %} {% tab title="SharePoint On-Premises" %} {% content-ref url="/pages/-LxevWzrcalReEzYJJ18" %} [NTLM Auth](/auth/strategies/ntlm) {% endcontent-ref %} {% content-ref url="/pages/-LxeviGxgiisbzKNDkOb" %} [ADFS Auth](/auth/strategies/adfs) {% endcontent-ref %} {% content-ref url="/pages/-LxevqUG1CnQAkiLp2g6" %} [FBA Auth](/auth/strategies/fba) {% endcontent-ref %} {% content-ref url="/pages/-LxevxbJMyGIKHDfukwN" %} [TMG Auth](/auth/strategies/tmg) {% endcontent-ref %} {% endtab %} {% endtabs %} ### Honorable mentions Within experimental in the [sandbox](https://github.com/koltyakov/gosip-sandbox/tree/master/strategies) these deserve mentioning: {% content-ref url="/pages/-LyP0uKumgGBYcmyUUc2" %} [On-Demand Auth](/auth/strategies/on-demand) {% endcontent-ref %} {% content-ref url="/pages/-LyEaWjPR9Xh0MyEteJX" %} [NTLM (alternative)](/auth/strategies/alternative-ntlm) {% endcontent-ref %} # Azure Certificate Auth Azure AD Certificate authentication This article is the sample showing Gosip custom auth with [AAD Certificate Authorization](https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization). ### Azure App registration 1\. Create or use existing app registration 2\. Make sure that the app is configured for a specific auth scenario: * Certificate Follow instructions: * O365 Admin -> Azure Active Directory * Generate self-signed certificate ```bash # PowerShell, run on a Windows machine $certName = "MyCert" $password = "MyPassword" $startDate = Get-Date $endDate = (Get-Date).AddYears(5) $securePass = (ConvertTo-SecureString -String $password -AsPlainText -Force) .\Create-SelfSignedCertificate.ps1 -CommonName $certName -StartDate $startDate -EndDate $endDate -Password $securePass ``` or on a Linux or macOS client via `openssl`: ```bash chmod +x ./Create-SelfSignedCertificate.sh ./Create-SelfSignedCertificate.sh ``` {% hint style="info" %} Get [scripts](https://github.com/koltyakov/gosip/tree/master/auth/azurecert). {% endhint %} * New App Registration * Accounts in this organizational directory only * API Permissions -> SharePoint :: Application :: Sites.FullControl.All -> Grant Admin Consent * Certificates & Secrets -> Upload `.cer` file ### JSON `private.json` sample: ```javascript { "siteUrl": "https://contoso.sharepoint.com/sites/test", "tenantId": "e4d43069-8ecb-49c4-8178-5bec83c53e9d", "clientId": "628cc712-c9a4-48f0-a059-af64bdbb4be5", "certPath": "cert.pfx", "certPass": "password" } ``` ### Usage sample ```go package main import ( "fmt" "log" "os" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/azurecert" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // TenantID: os.Getenv("AZURE_TENANT_ID"), // ClientID: os.Getenv("AZURE_CLIENT_ID"), // CertPath: os.Getenv("AZURE_CERTIFICATE_PATH"), // CertPass: os.Getenv("AZURE_CERTIFICATE_PASSWORD"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) res, err := sp.Web().Select("Title").Get() if err != nil { log.Fatal(err) } fmt.Printf("Site title: %s\n", res.Data().Title) } ``` # Azure Creds Auth Azure AD authorization with username and password This article is the sample showing Gosip custom auth with [AAD Username/Password Authorization](https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization). ### Azure App registration {% hint style="info" %} Follow the [steps](/auth/strategies/azure-environment-auth#azure-app-registration) {% endhint %} ### JSON `private.json` sample: ```javascript { "siteUrl": "https://contoso.sharepoint.com/sites/test", "tenantId": "e4d43069-8ecb-49c4-8178-5bec83c53e9d", "clientId": "628cc712-c9a4-48f0-a059-af64bdbb4be5", "username": "user@contoso.com", "password": "password" } ``` ### Usage sample ```go package main import ( "fmt" "log" "os" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/azurecreds" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // TenantID: os.Getenv("AZURE_TENANT_ID"), // ClientID: os.Getenv("AZURE_CLIENT_ID"), // Username: os.Getenv("AZURE_USERNAME"), // Password: os.Getenv("AZURE_PASSWORD"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) res, err := sp.Web().Select("Title").Get() if err != nil { log.Fatal(err) } fmt.Printf("Site title: %s\n", res.Data().Title) } ``` # Azure Env-based Auth Azure AD environment-based authentication ### Azure App registration 1\. Create or use existing app registration 2\. Make sure that the app is configured for a specific auth scenario: * Client credentials (*might not work with SharePoint but require a Certificate-based auth*) * **Certificate** * **Username/Password** (public clients flows must be enabled) * Managed identity Follow instructions: * O365 Admin -> Azure Active Directory * Generate self-signed certificate ```bash # PowerShell, run on a Windows machine $certName = "MyCert" $password = "MyPassword" $startDate = Get-Date $endDate = (Get-Date).AddYears(5) $securePass = (ConvertTo-SecureString -String $password -AsPlainText -Force) .\Create-SelfSignedCertificate.ps1 -CommonName $certName -StartDate $startDate -EndDate $endDate -Password $securePass ``` or on a Linux or macOS client via `openssl`: ```bash chmod +x ./Create-SelfSignedCertificate.sh ./Create-SelfSignedCertificate.sh ``` {% hint style="info" %} Get [scripts](https://github.com/koltyakov/gosip/tree/master/auth/azurecert). {% endhint %} * New App Registration * Accounts in this organizational directory only * API Permissions -> SharePoint :: Application :: Sites.FullControl.All -> Grant Admin Consent * Certificates & Secrets -> Upload `.cer` file * Use environment variables to provide creds bindings: * `AZURE_TENANT_ID` - Directory (tenant) ID in App Registration * `AZURE_CLIENT_ID` - Application (client) ID in App Registration * For certificate-base auth: * `AZURE_CERTIFICATE_PATH` - path to `.pfx` file * `AZURE_CERTIFICATE_PASSWORD` - password used for self-signed certificate * For username/password auth: * `AZURE_USERNAME` * `AZURE_PASSWORD` ### Auth configuration and usage ```go package main import ( "fmt" "log" "os" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip-sandbox/strategies/azureenv" ) func main() { // os.Setenv("AZURE_TENANT_ID", "b1bacba7-c38a-414b-8c8b-65df26a15749") // os.Setenv("AZURE_CLIENT_ID", "8ca10ce6-c3d5-47c6-b803-0ef3b619f464") // os.Setenv("AZURE_CERTIFICATE_PATH", "/path/to/cert.pfx") // os.Setenv("AZURE_CERTIFICATE_PASSWORD", "cert-password") authCnfg := &strategy.AuthCnfg{ SiteURL: os.Getenv("SPAUTH_SITEURL"), } client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) res, err := sp.Web().Select("Title").Get() if err != nil { log.Fatal(err) } fmt.Printf("Site title: %s\n", res.Data().Title) } ``` ### Environment variables auto-injection Environment variables can be automatically injected in a runtime for Azure AAD library. To use injection add correcponding environment variables in `private.json` into `env` JSON property: ```javascript { "siteUrl": "https://contoso.sharepoint.com/sites/site", "env": { "AZURE_TENANT_ID": "1efde0dc-21f5-4d3d-a053-1da762c7838c", "AZURE_CLIENT_ID": "7278fe9b-acd5-4be5-b688-999603560d31", "AZURE_CERTIFICATE_PATH": "./certs/MyCert.pfx", "AZURE_CERTIFICATE_PASSWORD": "MyPassword" } } ``` # Azure Device Flow Azure AD Device Token authentication This article is the sample showing Gosip custom auth with [AAD Device Token Authorization](https://docs.microsoft.com/en-us/azure/go/azure-sdk-go-authorization#use-device-token-authentication). > If you want users to sign in interactively, the best way is through device token authentication. This authentication flow passes the user a token to paste into a Microsoft sign-in site, where they then authenticate with an Azure Active Directory (AAD) account. This authentication method supports accounts that have multi-factor authentication enabled, unlike standard username/password authentication. ### Azure App registration 1\. Create or use existing app registration 2\. Make sure that the app is configured to support device flow * Authentication settings * Public client/native (mobile & desktop) * Suggested Redirect URIs for public clients (mobile, desktop) - - checked * Default client type - Yes - for Device code flow, [learn more](https://go.microsoft.com/fwlink/?linkid=2094804) * App permissions * Azure Service Management :: user\_impersonation * SharePoint :: based on your application requirements * etc. based on application needs ### Auth configuration and usage ```go package main import ( "fmt" "log" "os" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/device" ) func main() { authCnfg := &strategy.AuthCnfg{ SiteURL: os.Getenv("SPAUTH_SITEURL"), ClientID: os.Getenv("SPAUTH_AAD_CLIENTID"), TenantID: os.Getenv("SPAUTH_AAD_TENANTID"), } client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) res, err := sp.Web().Select("Title").Get() if err != nil { log.Fatal(err) } fmt.Printf("Site title: %s\n", res.Data().Title) } ``` When started the application interacts with user using device login. ```bash To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CL25ZF5N7 to authenticate. ``` After opening the link, providing device code and authenticating in browser the app is ready for communication with your SharePoint site. The strategy caches auth token in the context of the AAD ClientID. As a result, you won't see the sign in message. If it's not the desired behavior `.CleanTokenCache()` method can be called to clean the local cache. Note, that the technique is mostly applicable when user interaction is assumed. Usage of that auth approach in the headless scenarios is not the best as it can lead "stuck" application if no-one expects sign in interaction. # AddIn Only AddIn Only authentication {% hint style="danger" %} SharePoint Add-Ins will stop working for new tenants as of November 1st, 2024 and they will stop working for existing tenants and will be fully retired as of April 2nd, 2026. See [more](https://techcommunity.microsoft.com/t5/microsoft-sharepoint-blog/sharepoint-add-in-retirement-in-microsoft-365/ba-p/3982035). {% endhint %} ### Struct ```go type AuthCnfg struct { // SPSite or SPWeb URL, which is the context target for the API calls SiteURL string `json:"siteUrl"` // Client ID obtained when registering the AddIn ClientID string `json:"clientId"` // Client Secret obtained when registering the AddIn ClientSecret string `json:"clientSecret"` // Your SharePoint Online tenant ID (optional) Realm string `json:"realm"` } ``` Realm can be left empty or filled in, that will add small performance improvement. The easiest way to find tenant is to open SharePoint Online site collection, click `Site Settings` -> `Site App Permissions`. Taking any random app, the tenant ID (realm) is the GUID part after the `@`. See more details of [AddIn Configuration and Permissions](/auth/strategies/addin/configuration). ### JSON `private.json` sample: ```javascript { "siteUrl": "https://contoso.sharepoint.com/sites/test", "clientId": "e2763c6d-7ee6-41d6-b15c-dd1f75f90b8f", "clientSecret": "OqDSAAuBChzI+uOX0OUhXxiOYo1g6X7mjXCVA9mSF/0=" } ``` ### Code sample ```go package main import ( "log" // "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/addin" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // ClientID: os.Getenv("SPAUTH_CLIENTID"), // ClientSecret: os.Getenv("SPAUTH_CLIENTSECRET"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` ### Extending client secrets It's important to know that the legacy AddIn authentication's Client Secrets are issued for a limited time. After expiration, if not managed right way there is a risk to get a service connection aunothorized with the following message: AADSTS7000222: The provided client secret keys for app '\*\*\*' are expired. Visit the Azure portal to create new keys for your app: or consider using certificate credentials for added security: To renew an AddIn please follow [https://docs.microsoft.com/ru-ru/sharepoint/dev/sp-add-ins/replace-an-expiring-client-secret-in-a-sharepoint-add-in
]()or, long story short: ```powershell Install-Module -Name AzureAD Install-Module MSOnline Connect-MsolService # provide tenant admin account creds ``` ```powershell $clientId = 'e2763c6d-7ee6-41d6-b15c-dd1f75f90b8f' # replace with your clientId $bytes = New-Object Byte[] 32 $rand = [System.Security.Cryptography.RandomNumberGenerator]::Create() $rand.GetBytes($bytes) $rand.Dispose() $newClientSecret = [System.Convert]::ToBase64String($bytes) New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Symmetric -Usage Sign -Value $newClientSecret -StartDate (Get-Date) -EndDate (Get-Date).AddYears(1) New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Symmetric -Usage Verify -Value $newClientSecret -StartDate (Get-Date) -EndDate (Get-Date).AddYears(1) New-MsolServicePrincipalCredential -AppPrincipalId $clientId -Type Password -Usage Verify -Value $newClientSecret -StartDate (Get-Date) -EndDate (Get-Date).AddYears(1) $newClientSecret # outputs new clientSecret ``` ### Known issues AddIn Only auth is considered a legacy, in a production [Azure Cert](/auth/strategies/azure-certificate-auth) is vendor recommended. In new subscriptions you can face Grant App Permission disabled. You'll be getting the following error: ```json { "error": "invalid_request", "error_description": "Token type is not allowed." } ``` To enable this feature, connect to SharePoint using Windows PowerShell and then run: `set-spotenant -DisableCustomAppAuthentication $false`. ```powershell Install-Module -Name Microsoft.Online.SharePoint.PowerShell $adminUPN="" $orgName="" $userCredential = Get-Credential -UserName $adminUPN -Message "Type the password." Connect-SPOService -Url https://$orgName-admin.sharepoint.com -Credential $userCredential set-spotenant -DisableCustomAppAuthentication $false ``` # Configuration AddIn Configuration and Permissions For AddIn Only authentication to work register new addin within your SharePoint Online tenant. * Navigate to app registration page: https\://{organization}.sharepoint.com/sites/{site}/\_layouts/15/appregnew\.aspx * Click "Generate" button next to Client Id and Client Secret, fill in Title, App Domain, Redirect URI (you can type in any values you want).
* Copy Client Id and Client Secret and press "Create" button. * Apply permissions for the app on tenant or site collection level. #### Tenant scoped parmissions https\://{organization}-admin.sharepoint.com/\_layouts/15/appinv.aspx ```xml ``` #### Site collection scoped permissions https\://{organization}.sharepoint.com/sites/{site}/\_layouts/15/appinv.aspx ```xml ``` * Resolve addin by Client Id and paste in App's Permissions Request XML:
* Click "Create" and "Trust It".
To check which app principals are assigned for a site collection use: https\://{organization}.sharepoint.com/sites/{site}/\_layouts/15/appprincipals.aspx ### Disabled by default In new subscriptions you could be needed to enable Grant App Permission. Connect to SharePoint using Windows PowerShell and then run: `set-spotenant -DisableCustomAppAuthentication $false`. ```powershell Install-Module -Name Microsoft.Online.SharePoint.PowerShell $adminUPN="" $orgName="" $userCredential = Get-Credential -UserName $adminUPN -Message "Type the password." Connect-SPOService -Url https://$orgName-admin.sharepoint.com -Credential $userCredential set-spotenant -DisableCustomAppAuthentication $false ``` # NTLM Auth NTLM handshake authentication This type of authentication uses HTTP NTLM handshake in order to obtain authentication header. ### Struct ```go type AuthCnfg struct { // SPSite or SPWeb URL, which is the context target for the API calls SiteURL string `json:"siteUrl"` Domain string `json:"domain"` // AD domain name Username string `json:"username"` // AD user name Password string `json:"password"` // AD user password } ``` Gosip uses `github.com/Azure/go-ntlmssp` NTLM negotiator, however a custom one also can be [provided](https://github.com/koltyakov/gosip/issues/14) in case of demand. ### JSON `private.json` sample: ```javascript { "siteUrl": "https://www.contoso.com/sites/test", "username": "contoso\\john.doe", "password": "this-is-not-a-real-password" } ``` or ```javascript { "siteUrl": "https://www.contoso.com/sites/test", "username": "john.doe", "domain": "contoso", "password": "this-is-not-a-real-password" } ``` ### Code sample ```go package main import ( "log" // "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/ntlm" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // Username: os.Getenv("SPAUTH_USERNAME"), // Password: os.Getenv("SPAUTH_PASSWORD"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` If this strategy doesn't work in your environment yet you know for sure it's NTLM used try [this](/auth/strategies/alternative-ntlm) alternative. # NTLM (alternative) NTLM handshake authentication The default NTLM authentication uses [go-ntlmssp](https://github.com/Azure/go-ntlmssp) - the great package by Azure team. However, we found rare environments where it [fails](https://github.com/koltyakov/gosip/issues/14) for some reason. Fortunately, there is an [alternative](https://github.com/koltyakov/gosip-sandbox/tree/master/strategies/ntlm). ### Usage sample ```go package main import ( "fmt" "log" "os" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip-sandbox/strategies/ntlm" ) func main() { authCnfg := &strategy.AuthCnfg{ SiteURL: os.Getenv("SPAUTH_SITEURL"), Username: os.Getenv("SPAUTH_USERNAME"), Domain: os.Getenv("SPAUTH_DOMAIN"), Password: os.Getenv("SPAUTH_PASSWORD"), } client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) res, err := sp.Web().Select("Title").Get() if err != nil { log.Fatal(err) } fmt.Printf("Site title: %s\n", res.Data().Title) } ``` Use the alternative only as a fallback method if go-ntlmssp doesn't work and don't forget to post an issue to help making Open Source tools better. # On-Demand Auth Browser input interactive auth flow During the development, it's common to face a situation when production-level auth (AddIn Onli, Azure AD application) can't be configured in the desired timeframes and no auth strategies work. A simple example might be 2FA (multi-factor authentication) or custom ADFS provider. As a quick workaround, the On-Demand auth can help. On-Demand means that an interactive browser session is started where a user can provide the credentials as if he/she opens the SharePoint site and follows the same flow as reaching the site in a browser. In that strategy, the application actually opens the browser and communicates via debug protocol for the auth cookies when uses them in the requests. {% hint style="success" %} Check [On-Demand auth sources](https://github.com/koltyakov/gosip-sandbox/tree/master/strategies/ondemand) at GitHub. {% endhint %} On-Demand auth is based on [Lorca](https://github.com/zserge/lorca) project, however, a vital part of the [functionality](https://github.com/zserge/lorca/issues/46) is not exposed as a public API in Lorca, so the dependency is imported from a [fork](https://github.com/koltyakov/lorca) with only that small change in exposing one additional method. Lorca masters Chrome Debug Protocol, therefore, the Chrome/Chromium browser must be installed in the system where On-Demand auth is intended to be called. {% hint style="warning" %} Chrome is required for the strategy to work {% endhint %} ### Configure and usage sample ```go package main import ( "fmt" "log" "os" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip-sandbox/strategies/ondemand" ) func main() { authCnfg := &strategy.AuthCnfg{ SiteURL: os.Getenv("SPAUTH_SITEURL"), } client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) res, err := sp.Web().Select("Title").Get() if err != nil { log.Fatal(err) } fmt.Printf("Site title: %s\n", res.Data().Title) } ``` On-Demand configuration assumes only SiteURL to be provided as everything else is dynamically resolved while the transition to the browser page. The auth technique works for any strategy which is based on the cookies. The strategy caches the cookies in the context of the SharePoint host. As a result, you won't see the credentials prompt each time. If it's not the desired behavior `.CleanCookieCache()` method can be called to clean the local cache. Note, that the technique is only applicable when user interaction is assumed. Never ever use that auth approach in headless scenarios. # ADFS Auth User credentials authentication ### Struct ```go type AuthCnfg struct { // SPSite or SPWeb URL, which is the context target for the API calls SiteURL string `json:"siteUrl"` Username string `json:"username"` Password string `json:"password"` // Following are not required for SPO Domain string `json:"domain"` RelyingParty string `json:"relyingParty"` AdfsURL string `json:"adfsUrl"` AdfsCookie string `json:"adfsCookie"` } ``` See more details [ADFS user credentials authentication](https://github.com/s-kainet/node-sp-auth/wiki/ADFS-user-credentials-authentication). Gosip's ADFS also supports a scenario of ADFS or NTML behind WAP (Web Application Proxy) which adds additional auth flow and `EdgeAccessCookie` involved into play. ### JSON #### On-Premises configuration `private.json` sample: ```javascript { "siteUrl": "https://www.contoso.com/sites/test", "username": "john.doe@contoso.com", "password": "this-is-not-a-real-password", "relyingParty": "urn:sharepoint:www", "adfsUrl": "https://login.contoso.com", "adfsCookie": "FedAuth" } ``` #### On-Premises behing WAP configuration `private.json` sample: ```javascript { "siteUrl": "https://www.contoso.com/sites/test", "username": "john.doe@contoso.com", "password": "this-is-not-a-real-password", "relyingParty": "urn:AppProxy:com", "adfsUrl": "https://login.contoso.com", "adfsCookie": "EdgeAccessCookie" } ``` #### SharePoint Online configuration `private.json` sample: ```javascript { "siteUrl": "https://contoso.sharepoint.com/sites/test", "username": "john.doe@contoso.onmicrosoft.com", "password": "this-is-not-a-real-password" } ``` ### Code sample ```go package main import ( "log" // "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/adfs" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // Username: os.Getenv("SPAUTH_USERNAME"), // Password: os.Getenv("SPAUTH_PASSWORD"), // /* other auth props ...*/ // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` # FBA Auth Form-based authentication for SharePoint On-Premises ### Struct ```go type AuthCnfg struct { // SPSite or SPWeb URL, which is the context target for the API calls SiteURL string `json:"siteUrl"` // Username for SharePoint On-Prem, // format depends in FBA settings, can include domain or doesn't Username string `json:"username"` // User password Password string `json:"password"` } ``` ### JSON `private.json` sample: ```javascript { "siteUrl": "https://www.contoso.com/sites/test", "username": "john.doe", "password": "this-is-not-a-real-password" } ``` ### Code sample ```go package main import ( "log" // "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/fba" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // Username: os.Getenv("SPAUTH_USERNAME"), // Password: os.Getenv("SPAUTH_PASSWORD"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` # TMG Auth Microsoft Forefront Threat Management Gateway Currently is legacy but was a popular way of exposing SharePoint into external world back in the days. ### Struct ```go type AuthCnfg struct { // SPSite or SPWeb URL, which is the context target for the API calls SiteURL string `json:"siteUrl"` // Username for SharePoint On-Prem, // format depends in TMG settings, can include domain or doesn't Username string `json:"username"` // User password Password string `json:"password"` } ``` ### JSON `private.json` sample: ```javascript { "siteUrl": "https://www.contoso.com/sites/test", "username": "john.doe", "password": "this-is-not-a-real-password" } ``` ### Code sample ```go package main import ( "log" // "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/tmg" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // Username: os.Getenv("SPAUTH_USERNAME"), // Password: os.Getenv("SPAUTH_PASSWORD"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` # SAML Auth \[Deprecated] SharePoint Online user credentials authentication ⚠️ **Microsoft retired this authentication mechanism** This authentication option uses Microsoft Online Security Token Service `https://login.microsoftonline.com/extSTS.srf` and SAML tokens in order to obtain authentication cookie. ### Struct ```go type AuthCnfg struct { // SPSite or SPWeb URL, which is the context target for the API calls SiteURL string `json:"siteUrl"` // Username for SharePoint Online, e.g. `[user]@[company].onmicrosoft.com` Username string `json:"username"` // User or App password Password string `json:"password"` } ``` ### JSON `private.json` sample: ```javascript { "siteUrl": "https://contoso.sharepoint.com/sites/test", "username": "john.doe@contoso.onmicrosoft.com", "password": "this-is-not-a-real-password" } ``` ### Code sample ```go package main import ( "log" // "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/saml" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // Username: os.Getenv("SPAUTH_USERNAME"), // Password: os.Getenv("SPAUTH_PASSWORD"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` # Anonymous No-auth mode It's not an auth strategy but a mode without any authentication flow applied to the SPClient. Anonymous mode can be handy in a situation when Gosip SharePoint-aware helpers intended to be used however authentication is handled by any other middleware. ### Struct Only `SiteURL` is required. ```go type AuthCnfg struct { // SPSite or SPWeb URL, which is the context target for the API calls SiteURL string `json:"siteUrl"` } ``` ### JSON `private.json` sample: ```javascript { "siteUrl": "https://www.contoso.com/sites/test" } ``` ### Code sample ```go package main import ( "log" // "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/anon" ) func main() { // authCnfg := &strategy.AuthCnfg{ // SiteURL: os.Getenv("SPAUTH_SITEURL"), // } // or using `private.json` creds source authCnfg := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` # Dynamic auth Resolving a strategy dynamically in runtime When you deal with multiple SharePoint environments with different strategies and concidering same code automatically resolving a strategy use the dynamic auth helper. With the dynamic auth you'd need extending a `private.json` content with `strategy` property, containing a name of a strategy. E.g.: ```json { "strtegy": "addin", "siteUrl": "https://contoso.sharepoint.com/sites/site", "clientId": "...", "clientSecret": "..." } ``` Available strategy names are: azurecert, azurecreds, device, addin, adfs, fba, ntlm, saml, tmg. ### Usage ```go package main import ( "flag" "log" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" "github.com/koltyakov/gosip/auth" ) func main() { config := flag.String("config", "./config/private.json", "Config path") flag.Parse() authCnfg, err := auth.NewAuthFromFile(*config) if err != nil { log.Fatalf("unable to get config: %v", err) } // or // authCnfg, _ := NewAuthFromFile(strategyName) // _ = auth.ParseConfig(byteJsonCreds) client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) // ... } ``` With a custom auth involved it can be extended like [this](https://github.com/koltyakov/cq-source-sharepoint/blob/main/resources/auth/auth.go). # Custom Auth Custom authentication mechanisms Gosip allows providing custom authentication mechanisms. For example, you are considering reusing Fluent API helpers and HTTP Client but existing authentication strategies do not feet your environment specifics. Maybe your tenant configured with custom ADFS provider, maybe it's 2FA and there are no alternatives and you need On-Demand auth, but it missed in Gosip strategies list? Fortunately, this is not any sort of stopper. All included authentication strategies are a sort of a pluging and it's rather affordable to add a new strategy on your own. Let's take a look at any strategy binding: ```go package main import ( "log" "os" "github.com/koltyakov/gosip" strategy "github.com/koltyakov/gosip/auth/saml" ) func main() { authCnfg := &strategy.AuthCnfg{ SiteURL: os.Getenv("SPAUTH_SITEURL"), Username: os.Getenv("SPAUTH_USERNAME"), Password: os.Getenv("SPAUTH_PASSWORD"), } client := &gosip.SPClient{AuthCnfg: authCnfg} // use client in raw requests or bind it with Fluent API ... } ``` What we can see? Some strategy is imported into the `strategy` namespace. A strategy has `AuthCnfg` struct with some public properties which are obviously taking place in authentication flow. This struct is then passed to `&gosip.SPClient{AuthCnfg: authCnfg}` and somehow after following binding the requests are authenticated. For this construction to work `strategy.AuthCnfg` should implement `gosip.AuthCnfg` interface which is: ```go // AuthCnfg is an abstract auth config interface, // allows different authentications strategies' dependency injection type AuthCnfg interface { // Authentication middleware fabric // applyes round tripper or enriches requests with authentication and metadata SetAuth(req *http.Request, client *SPClient) error // Authentication initializer (token/cookie/header, expiration, error) // to support cabability for exposing tokens for external tools // e.g. as of this sample project https://github.com/koltyakov/spvault GetAuth() (string, int64, error) ParseConfig(jsonConf []byte) error // Parses credentials from a provided JSON byte array content ReadConfig(configPath string) error // Reads credentials from storage GetSiteURL() string // SiteURL getter method GetStrategy() string // Strategy code getter } ``` Philosophy of the strategies is to have two initiation modes, the first is a strict declaration of the creds and the second one is reading credentials from the config. That config is not necessarily a file on the file system it can be a request to a key vault or OS credential manager, etc. As the interface is passed to `gosip.SPClient` struct, Gosip knows nothing about the creds and the context, for that reason `GetSiteURL` method is vital to target requests to a correct root URL. `GetStrategy` method should return the string alias value of the strategy name if something specific should be happening based on its value. `GetAuth` method is for token and cookie-based authentications, it can be omitted and return just a blank value, or it can be an actual place for authentication flow happening inside, returning a cached string which is when applied somehow to the requests making them authenticated. In case of custom logic, we'd recommend using GetAuth method and don't forget TTL caching to reduce roundtrips. With a robust external auth client, GetAuth can be dummy minimum ([NTLM example](https://github.com/koltyakov/gosip/blob/915876dc1a96ddb8db5b8ed7c403a92608db3f05/auth/ntlm/helpers.go#L15) shows this approach). And finally, `SetAuth`, the method where all the magic happening. `SetAuth` method is a middleware, it receives runtime request and should append authentication stuff. Check these as samples: [NTML's SetAuth](https://github.com/koltyakov/gosip/blob/915876dc1a96ddb8db5b8ed7c403a92608db3f05/auth/ntlm/auth.go#L109), [cookie-based auth](https://github.com/koltyakov/gosip/blob/915876dc1a96ddb8db5b8ed7c403a92608db3f05/auth/saml/auth.go#L94), [Bearer token-based auth](https://github.com/koltyakov/gosip/blob/915876dc1a96ddb8db5b8ed7c403a92608db3f05/auth/addin/auth.go#L96). By implementing AuthCnfg struct and gosip.AuthCnfg interface any custom authentication can be added to Gosip. {% content-ref url="/pages/-LyP0uKumgGBYcmyUUc2" %} [On-Demand Auth](/auth/strategies/on-demand) {% endcontent-ref %} {% content-ref url="/pages/-LyEaWjPR9Xh0MyEteJX" %} [NTLM (alternative)](/auth/strategies/alternative-ntlm) {% endcontent-ref %} # HTTP Client 🔨 Provides low-level communication with any SharePoint API Gosip HTTP client for SharePoint allows consuming any HTTP resource or API, it can even bypass SharePoint site pages (with all assets) within a dev toolchain ([proxy, dev server](https://github.com/koltyakov/gosip/tree/master/cmd/samples/proxy)). Gosip HTTP client is SharePoint nuances-aware, it takes care under the hood of such things as headers, API calls retries, threshholds, error handling, POST API requests Digests, and, of course, authentication and its renewal. However, dealing at low-level, means you should know SharePoint API rather well and you're ok with the verbosity of Go. If you just starting with SharePoint please consider [Fluent API](/api/fluent) client first and HTTP client for none covered methods and custom stuff. ### Usage ```go package main import ( "fmt" "log" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/ntlm" ) func main() { auth := &strategy.AuthCnfg{} configPath := "./config/private.json" if err := auth.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v\n", err) } spClient := api.NewHTTPClient(&gosip.SPClient{AuthCnfg: auth}) endpoint := auth.GetSiteURL() + "/_api/web?$select=Title" data, err := spClient.Get(endpoint, nil) if err != nil { log.Fatalf("%v\n", err) } // spClient.Post(endpoint, body, nil) // generic POST // generic DELETE helper crafts "X-Http-Method"="DELETE" header // spClient.Delete(endpoint, nil) // generic UPDATE helper crafts "X-Http-Method"="MERGE" header // spClient.Update(endpoint, body, nil) // CSOM helper (client.svc/ProcessQuery) // spClient.ProcessQuery(endpoint, body, nil) fmt.Printf("response: %s\n", data) } ``` ### Methods HTTP Client methods covers those which used in SharePoint all the way. In a contract to default Go `http.Client`'s `.Do` method, Gosip HTTP Client's methods proceed and read response body to return array of bytes. Which reduce amount of scaffolded code a bit and more handy for the REST API consumption, in our opinion, which is 90% of use cases. #### Get Sends `GET` request, embeds `"Accept": "application/json;odata=verbose"` header as a default. Use it for REST API `GET` calls. #### Post Sends `POST` request, embeds `"Accept": "application/json;odata=verbose"` and `"Content-Type": "application/json;odata=verbose;charset=utf-8"` headers as default. X-RequestDigest is received, cached, and embed automattically. Use it for REST API `POST` calls. #### Delete Sends `POST` request, embeds `"Accept": "application/json;odata=verbose"`, `"Content-Type": "application/json;odata=verbose;charset=utf-8"`, and `"X-Http-Method": "DELETE"` headers as default. X-RequestDigest is received, cached, and embed automattically. Use it for REST API `POST` calls with delete resource intention. #### Update Sends `POST` request, embeds `"Accept": "application/json;odata=verbose"`, `"Content-Type": "application/json;odata=verbose;charset=utf-8"`, and `"X-Http-Method": "MERGE"` headers as default. X-RequestDigest is received, cached, and embed automattically. Use it for REST API `POST` calls with update resource intention. ### ProcessQuery Sends `POST` request to `/_vti_bin/client.svc/ProcessQuery` endpoint (CSOM). All required headers for a CSOM request are embed automatically. Method's body should stand for a valid CSOM XML package. The response body is parsed for error handling, yet returned in it's original form. ### Low-level HTTP client Sometimes more control is required, e.g. when downloading files or large responses you could prefer precessing a response in chunks. This can be achieved with the low-level usage of Gosip HTTP client. ```go client := &gosip.SPClient{AuthCnfg: auth} var req *http.Request // Initiate API request // ... resp, err := client.Execute(req) if err != nil { fmt.Printf("Unable to request api: %v", err) return } ``` SPClient has `Execute` method which is a wrapper function injecting SharePoint authentication and ending up calling http.Client's `Do` method. So you can dive down to native `*http.Request` at this point it's just a standard request from "net/http" package, but authenticated to SharePoint and with some batteries under the hood for a seemles API integration. # Fluent API 🏄 Fluent, chainable, IntelliSense powered syntax to master SharePoint API Provides a simple way of constructing API endpoint calls with IntelliSense and chainable syntax. ![](/files/-LxfjVQYYo0KGmfmLQJq) ### Usage sample ```go package main import ( "encoding/json" "fmt" "log" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/addin" ) func main() { // Getting auth params and client client, err := getAuthClient() if err != nil { log.Fatalln(err) } // Binding SharePoint API sp := api.NewSP(client) // Custom headers (optional) headers := map[string]string{ "Accept": "application/json;odata=minimalmetadata", "Accept-Language": "de-DE,de;q=0.9", } config := &api.RequestConfig{Headers: headers} // Chainable request sample data, err := sp.Conf(config).Web().Lists().Select("Id,Title").Get() if err != nil { log.Fatalln(err) } // Response object unmarshalling // struct depends on OData mode and API method res := &struct { Value []struct { ID string `json:"Id"` Title string `json:"Title"` } `json:"value"` }{} if err := json.Unmarshal(data, &res); err != nil { log.Fatalf("unable to parse the response: %v", err) } for _, list := range res.Value { fmt.Printf("%+v\n", list) } } func getAuthClient() (*gosip.SPClient, error) { configPath := "./config/private.spo-addin.json" auth := &strategy.AuthCnfg{} if err := auth.ReadConfig(configPath); err != nil { return nil, fmt.Errorf("unable to get config: %v", err) } return &gosip.SPClient{AuthCnfg: auth}, nil } ``` ### Main concepts * Get authenticated * Construct root `SP` object using `api.NewSP(client)` * Construct API calls in a fluent way * Parse responses in the Go way * Embrase strongly typed generic responses * Build awesome apps in Go for SharePoint # Hooks Request events handlers Gosip provides an events system with a set of handlers that can be optionally defined to track different client communication aspects such as request tracking, retries and response error logging, to name just a few. To define the handlers `Hooks` object should be configured and passed to `gosip.SPClient` struct. ```go var authCnfg gosip.AuthCnfg // ... auth config initiation is omitted &gosip.SPClient{ AuthCnfg: authCnfg, Hooks: &gosip.HookHandlers{ // handlers function definition }, } ``` The following handlers are available at the moment: ```go // HookHandlers struct to configure events handlers type HookHandlers struct { OnError func(event *HookEvent) // when error appeared OnRetry func(event *HookEvent) // before retry request OnRequest func(event *HookEvent) // before request is sent OnResponse func(event *HookEvent) // after response is received } ``` All of the handlers are optional. A handler receives `HookEvent` pointer which contains request pointer, response status code, and error (if applicable for an event), and time information to track duration since a request started an event happened. Hooks sample: ```go // Define requests hook handlers client.Hooks = &gosip.HookHandlers{ OnError: func(e *gosip.HookEvent) { fmt.Println("\n======= On Error ========") fmt.Printf(" URL: %s\n", e.Request.URL) fmt.Printf(" StatusCode: %d\n", e.StatusCode) fmt.Printf(" Error: %s\n", e.Error) fmt.Printf(" took %f seconds\n", time.Since(e.StartedAt).Seconds()) fmt.Printf("=========================\n\n") }, OnRetry: func(e *gosip.HookEvent) { fmt.Println("\n======= On Retry ========") fmt.Printf(" URL: %s\n", e.Request.URL) fmt.Printf(" StatusCode: %d\n", e.StatusCode) fmt.Printf(" Error: %s\n", e.Error) fmt.Printf(" took %f seconds\n", time.Since(e.StartedAt).Seconds()) fmt.Printf("=========================\n\n") }, OnRequest: func(e *gosip.HookEvent) { if e.Error == nil { fmt.Println("\n====== On Request =======") fmt.Printf(" URL: %s\n", e.Request.URL) fmt.Printf(" auth injection took %f seconds\n", time.Since(e.StartedAt).Seconds()) fmt.Printf("=========================\n\n") } }, OnResponse: func(e *gosip.HookEvent) { if e.Error == nil { fmt.Println("\n====== On Response =======") fmt.Printf(" URL: %s\n", e.Request.URL) fmt.Printf(" StatusCode: %d\n", e.StatusCode) fmt.Printf(" took %f seconds\n", time.Since(e.StartedAt).Seconds()) fmt.Printf("==========================\n\n") } }, } ``` Hooks can be handy for global logging streaming and metrics collection. It is recommended using asynchronous and only lightweight logic inside hooks. # Retries Requests retries on error statuses Gosip HTTP client is preconfigured with retry policies so different kinds of failures can be back off. Current implementation of policies assumes a specific number of retries for a specific response HTTP Status Code. One would only consider retries for "non-ok" status codes and only those which represents server or networking error which has chances for backing off on a next try. ### Defaults The default policies are: | StatusCode | Retries | Description | | ---------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 401 | 5 | Unauthorized. A retry might help if apply authentication restores an overdue token or handshake. | | 429 | 5 | Too many requests throttling error response. In the case of API throttling (relevant for SharePoint Online), retrying is also aware of Retry-After header for delay detection. | | 500 | 1 | Internal Server Error. Rarely can be restored. | | 503 | 10 | Service Unavailable. Fixes intermittent issues with the service. | | 504 | 5 | Gateway Timeout Error. | Retries' delay increases with each attempt: 200ms, 400ms, 800ms, 1.6s, 3.2s, and so on using the following progression formula: ```go time.Duration(100*math.Pow(2, float64(retry))) * time.Millisecond ``` For the responses with `Retry-After` header, retry after value is used in preference as threshold is usually require longer wait until next request permitted. ### Custom policy A custom policy can be provided on demand in the following way: ```go var authCnfg gosip.AuthCnfg // ... auth config initiation is omitted &gosip.SPClient{ AuthCnfg: authCnfg, RetryPolicies: map[int]int{ // merged with default policies 500: 2, // overwrites default 503: 5, // overwrites default 401: 0, // disables default }, } ``` ### Disabling retries for a specific request When a specific request assumes no retries, e.g. resolving a folder which potentially doesn't exist but you know that 500 is returned in case of failure and it's a faster algorithm to try optimistically and check or create on an error only, it can be handy disabling any retries but only scoped to a specific request or series of requests. For that purposes `X-Gosip-NoRetry` header is the resolution. ```go headers := map[string]string{ "X-Gosip-NoRetry": "true", } conf := &RequestConfig{ Headers: headers, } if _, err := sp.Web().GetFolder(folderURI).Conf(conf).Get(); err != nil { fmt.Println(err) } ``` ### Custom retry delays There is no direct way to redefine delays between retries, there was no such demand. However, extending delays is achievable via [OnRetry hook](/api/hooks). The `OnRetry` hook is called after it's known that a retry is needed but before the actual retry. The mechanics of retry checks is implemented in the way that `should retry check` actually also waits for the delay. In other words, `OnRetry` hook is fired right before the next retry is sent. So, if that hook is locked for a time the delay is increased correspondingly. That might be helpful for adding extra delay, yet won't work for the cases when the delays should be reduced or flatten. While adding timeouts in hooks, make sure you're using something respecting context cancelation: ```go sleepTimeout := 1 * time.Second select { case <-e.Request.Context().Done(): return case <-time.After(sleepTimeout): } ``` # Context Using Go context with SP client Gosip client respects native Go [Context](https://golang.org/pkg/context), you can pass a context on a low level or to a Fluent API to control requests's deadlines, cancellation signals, and other request-scoped values across API boundaries and between processes. ### Low level client On the [low level](/api/http-client#low-level-http-client) dealing with the contexts is identical to native approach: ```go client := &gosip.SPClient{AuthCnfg: auth} var req *http.Request // Initiate API request // ... req = req.WithContext(context.Background()) // <- pass a context resp, err := client.Execute(req) if err != nil { fmt.Printf("Unable to request api: %v", err) return } ``` ### HTTP Client While using HTTPClient, context is defined together with request config. ```go spClient := api.NewHTTPClient(&gosip.SPClient{AuthCnfg: auth}) endpoint := auth.GetSiteURL() + "/_api/web?$select=Title" reqConf := &api.RequestConfig{ Context: context.Background(), // <- pass a context } data, err := spClient.Get(endpoint, reqConf) if err != nil { log.Fatalf("%v\n", err) } // spClient.Post(endpoint, body, reqConf) // generic POST // generic DELETE helper crafts "X-Http-Method"="DELETE" header // spClient.Delete(endpoint, reqConf) // generic UPDATE helper crafts "X-Http-Method"="MERGE" header // spClient.Update(endpoint, body, reqConf) // CSOM helper (client.svc/ProcessQuery) // spClient.ProcessQuery(endpoint, body, reqConf) ``` ### Fluent API With Fluent API, context is managed in the same way as in the previous example with the only difference how it is chained to fluent syntax. ```go config := &api.RequestConfig{ Context: context.Background(), // <- pass a context } sp := api.NewSP(client).Conf(config) data, err := sp.Web().Lists().Select("Id,Title").Get() if err != nil { log.Fatalln(err) } ``` `Conf` method can be used almost on any hierarchy level. It's inherited with capability to redefine. # Library Initiation Configuring authentication and API client ## Binding authentication and API client ### Understand SharePoint environment type and authentication strategy Based on authentication provider supported and configured in your SharePoint Farm environment different library [authentication strategies](/auth/strategies) might be or not be applicable. If you have no idea which strategy is used within your farm the question is better be addressed to SharePoint admins. Let's assume it's SharePoint Online and Add-In Only permissions. Then `strategy "github.com/koltyakov/gosip/auth/addin"` sub package should be used. ```go package main import ( "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/addin" ) ``` It could have been SharePoint On-Premise and NTLM and `strategy "github.com/koltyakov/gosip/auth/ntlm"` as imported strategy. ### Initiate authentication object Different authentication strategies assumes different credential parameters. Sometimes it can be Username/Password, sometimes CertPath or ClientID/ClientSecret. Please refer a specific strategy documentation for relevalt for the auth type parameters. Credential can be passed directly in AuthCnfg's struct. Here a sample for `addin`: ```go auth := &strategy.AuthCnfg{ SiteURL: os.Getenv("SPAUTH_SITEURL"), ClientID: os.Getenv("SPAUTH_CLIENTID"), ClientSecret: os.Getenv("SPAUTH_CLIENTSECRET"), } ``` An anternative is using a configuration file (see [more](/auth/overview#authentication-strategies)). Which is a JSON containing the same parameters as used with AuthCnfg's struct. ```go configPath := "./config/private.json" auth := &strategy.AuthCnfg{} err := auth.ReadConfig(configPath) if err != nil { fmt.Printf("Unable to get config: %v\n", err) return } ``` ### Bind auth client with Fluent API Now when auth is bound it should be passed to client and Fluent API instance: ```go client := &gosip.SPClient{AuthCnfg: auth} sp := api.NewSP(client) ``` Most of the samples starts with `sp` assuming configuration described above is already in place. ```go res, err := sp.Web().Select("Title").Get() if err != nil { fmt.Println(err) } fmt.Printf("%s\n", res.Data().Title) ``` # Basic CRUD Create, read, update and delete CRUD is the most commonly requested type of API consumption. This example demonstrate basic operations on a list items sample. Hovewer, SharePoint has a variety of nuances even when it comes to just getting items from a list. ### Getting list object ```go // The recommended way of getting lists is by using their relative URIs // can be a short form without full web relative URL prefix list := sp.Web().GetList("Lists/MyList") // other common but less recommended way of getting a list is // list := sp.Web().Lists().GetByTitle("My List") ``` ### Getting items ```go // itemsResp is a byte array read from response body // powered with some processing "batteries" as Data() or Unmarshal() methods itemsResp, err := list.Items(). Select("Id,Title"). // OData $select modifier, limit what props are retrieved OrderBy("Id", true). // OData $orderby modifier, defines sort order Top(10). // OData $top modifier, limits page size Get() // Finalizes API constructor and sends a response if err != nil { log.Fatal(err) } // Data() method is a helper which unmarshals generic structure // use custom structs and unmarshal for custom fields for _, item := range itemsResp.Data() { itemData := item.Data() fmt.Printf("ID: %d, Title: %s\n", itemData.ID, itemData.Title) } ``` Based on a use case, items can be reveived with alternative methods, as `.GetAll`, `.GetPaged`, `.GetByCAML`, or even lists methods as `.RenderListData`. ### Adding an item ```go // Payload should be a valid JSON stringified string converted to byte array // Payload's properties must be a valid OData entity types itemPayload := []byte(`{ "Title": "New Item" }`) // Constructs and sends add operation request // itemAddRes is a byte array read from response body with extra methods itemAddRes, err := list.Items().Add(itemPayload) if err != nil { log.Fatal(err) } fmt.Printf("Raw response: %s\n", itemAddRes) fmt.Printf("Added item's ID: %d\n", itemAddRes.Data().ID) ``` Payloads can be constructed in a usual Go way using marshalling struct or string maps to JSON. E.g.: ```go itemMetadata := &struct{ Title string `json:"Title"` }{ Title: "New Item", } itemPayload, _ := json.Marshal(itemMetadata) ``` or: ```go itemMetadata := map[string]interface{}{ "Title": "New Item", } itemPayload, _ := json.Marshal(itemMetadata) ``` up to your preferences. ### Getting a specific item ```go itemID := 42 // should specific item ID itemRes, err := list.Items().GetByID(itemID).Get() if err != nil { log.Fatal(err) } fmt.Printf("Raw response: %s\n", itemRes) fmt.Printf("Item metadata: %+v\n", itemRes.Data()) ``` ### Updating an item ```go // Payload should be a valid JSON stringified string converted to byte array // Payload's properties must be a valid OData entity types itemUpdatePayload := []byte(`{ "Title": "Updated Title" }`) itemID := 42 // should specific item ID // Constructs and sends update operation request // itemUpdateRes is a byte array read from response body with extra methods itemUpdateRes, err := list.Items().GetById(itemID).Update(itemUpdatePayload) if err != nil { log.Fatal(err) } fmt.Printf("Item is updated, %s\n", itemUpdateRes.Data().Modified) ``` Payloads can be constructed in a usual Go way using marshalling struct or string maps to JSON, same as in add operations. ### Delete an item Item can be not only deleted but recycled with a further restore operation, which can be provide more safety. ```go itemID := 42 // should specific item ID // list.Items().GetByID(itemID).Delete() // or .Recycle() if _, err := list.Items().GetByID(itemID).Recycle(); err != nil { log.Fatal(err) } ``` # Documents Download & upload files from/to SharePoint is simple SharePoint is ECM (Enterprise Content Management) system and it's common to expect files being uploaded, downloaded, migrated, processes, and managed in a variety ways. Gosip provides an easy way of dealing with SharePoint document listaries, files and folders. ### Getting library object Document library in SharePoint is almost the same as a List, but with intention of being a container for files. ```go // The recommended way of getting lists is by using their relative URIs // can be a short form without full web relative URL prefix list := sp.Web().GetList("MyLibrary") // other common but less recommended way of getting a list is // list := sp.Web().Lists().GetByTitle("My Library") ``` ### Getting folder object ```go folder := sp.Web().GetFolder("MyLibrary/Folder01") ``` ### Getting file object ```go file := sp.Web().GetFile("MyLibrary/Folder01/File01.txt") ``` ### Adding new folder ```go // folderResp is a byte array read from response body with extra methods folderResp, err := folder.Folders().Add("New Folder Name") if err != nil { log.Fatal(err) } fmt.Printf("New folder URL: %s\n", folderResp.Data().ServerRelativeURL) ``` ### Deleting folders ```go folderRelativeURL := "MyLibrary/Folder01/New Folder Name" if _, err := sp.Web().GetFolder(folderRelativeURL).Delete(); err != nil { log.Fatal(err) } ``` ### Adding/uploading a file ```go // fileAddResp is a byte array read from response body with extra methods fileAddResp, err := folder.Files(). Add("My File.txt", []byte("File content"), true) if err != nil { log.Fatal(err) } fmt.Printf("New file URL: %s\n", fileAddResp.Data().ServerRelativeURL) ``` Obviously, file content can be a result of reading a file from disk, e.g.: ```go content, err := ioutil.ReadFile("/path/to/file.txt") if err != nil { log.Fatal(err) } fileAddResp, err := folder.Files().Add("My File.txt", content, true) if err != nil { log.Fatal(err) } ``` For the large files it's better using AddChunked API, hovewer, it was not available in SharePoint 2013. ```go file, err := os.Open("/path/to/large/file.zip") if err != nil { log.Fatalf("unable to read a file: %v\n", err) } defer file.Close() fileAddResp, err := folder.Files().AddChunked("My File.zip", file, nil) if err != nil { log.Fatal(err) } fmt.Printf("New file URL: %s\n", fileAddResp.Data().ServerRelativeURL) ``` [More details](/samples/chunk-upload). ### Downloading files ```go fileRelativeURL := "MyLibrary/Folder01/File01.txt" data, err := sp.Web().GetFile(fileRelativeURL).Download() if err != nil { log.Fatal(err) } file, err := os.Create("/path/toLocal/file.txt") if err != nil { log.Fatalf("unable to create a file: %v\n", err) } defer file.Close() _, err = file.Write(data) if err != nil { log.Fatalf("unable to write to file: %v\n", err) } file.Sync() ``` For a large files it's better getting file reader through: ```go fileRelativeURL := "MyLibrary/Folder01/File01.txt" fileReader, err := sp.Web().GetFile(fileRelativeURL).GetReader() if err != nil { log.Fatal(err) } defer fileReader.Close() file, err := os.Create("/path/toLocal/file.txt") if err != nil { log.Fatalf("unable to create a file: %v\n", err) } defer file.Close() if _, err := io.Copy(file, fileReader); err != nil { log.Fatalf("unable to save a file: %v\n", err) } ``` ### Summary Using Gosip you can concentrate on business logic and Go language aspects while processing documents actions in SharePoint seemlessly. With use of IntelliSense and Fluent syntax other supported actions can be consumed based on a specific requirement. # Chunk upload Uploading files in chunks For large documents it's better using files Chunk API which allows splitting upload into separate REST API requests reducing memory consumption and providing a more manageable upload mechanism. Files Chunk API appeared in SharePoint 2016 and should not be expected to work in the previous version. {% hint style="warning" %} Chunk upload can't be used in SharePoint 2013 {% endhint %} ### Basic chunk upload example ```go file, err := os.Open("/path/to/large/file.zip") if err != nil { log.Fatalf("unable to read a file: %v\n", err) } defer file.Close() fileAddResp, err := foler.Files().AddChunked("My File.zip", file, nil) if err != nil { log.Fatal(err) } fmt.Printf("New file URL: %s\n", fileAddResp.Data().ServerRelativeURL) ``` ### Upload flow Fluent API abstracts some aspects of the chunked upload. Internally, `/StartUpload`, `/ContinueUpload`, `/CancelUpload`, and `/FinishUpload` endpoints are used. However, these are not exposed for the simplicity. Nonetheless, there is a way for canceling upload with the help of `api.AddChunkedOptions`. ### Add chunked options There are a few options which are optional. The third method parameter is a configuration structure. ```go type AddChunkedOptions struct { Overwrite bool // should overwrite existing file // on progress callback, execute custom logic on each chunk // if the Progress is used it should return "true" to continue upload // otherwise upload is canceled Progress func(data *FileUploadProgressData) bool ChunkSize int // chunk size in bytes } ``` Defaults are: `Overwrite` is true and `ChunkSize` equals to 10485760 bytes. `Progress` callback allows not only trigger a progress logic, for example upload percentage update, but also to cancel an upload. To cancel an upload the progress callback should return `false`. ```go filePath := "/path/to/large/file.zip" file, err := os.Open(filePath) if err != nil { log.Fatalf("unable to read a file: %v\n", err) } defer file.Close() info, _ := os.Stat(filePath) options := &api.AddChunkedOptions{ Overwrite: false, ChunkSize: 5 * 1024 * 1024, Progress: func(data *api.FileUploadProgressData) bool { fmt.Printf( "Block %d, sent %d%%\n", data.BlockNumber, 100*data.FileOffset/fi.Size(), ) return true // to cancel an upload on some external condition, return "false" }, } if _, err := foler.Files().AddChunked("My File.zip", file, options); err != nil { log.Fatal(err) } ``` # Permissions Managing roles and objects permissions Permissions management is an important part of business logic which can be met in worker processes and workflows. Let's explore how Gosip Fluent API wraps up SharePoint securable object and permissions. First of all, by securable objects mean any SharePoint artifacts which can be configured with unique permissions: webs, lists, libraries, items, etc. Fluent API scopes role operations under `.Roles()` method. ### Checking is an object has unique permissions ```go item := list.Items().GetByID(1) hasUniqueAssignments, err := item.Roles().HasUniqueAssignments() if err != nil { log.Fatal(err) } fmt.Printf("Has unique permissions: %t\n", hasUniqueAssignments) ``` ### Breaking role inheritance To assign unique permissions to an object its role inheritence must be broken. ```go item := list.Items().GetByID(1) if err := item.Roles().BreakInheritance(true, false); err != nil { log.Fatal(err) } // where the first argument stands for `copyRoleAssigments` // - if true the permissions are copied from the current parent scope // second argument is `clearSubScopes` // - true to make all child securable objects inherit role assignments // from the current object ``` Be aware that abusing breaking role inheritence and having too many unique permissions can have a detrimental effect on SharePoint performance. The architecture of a solution which involves many unique permissions has too be planned carefully. See some valuable [recommendations](https://docs.microsoft.com/en-us/sharepoint/sites/best-practices-for-using-fine-grained-permissions-in-sharepoint-server). ### Role definitions Before assigning permissions it's important to know how to get role definitions. > A role definition is a collection of rights bound to a specific object. Role definitions (for example, Full Control, Read, Contribute, Design, or Limited Access) are scoped to the Web site and mean the same thing everywhere within the Web site, but their meanings can differ between sites within the same site collection. Role definitions can also be inherited from the parent Web site, just as permissions can be inherited. Definitions are csoped as Web level in `.RoleDefinitions()` quariable collection. ```go def, err := sp.Web().RoleDefinitions().GetByType(api.RoleTypeKinds.Contributor) if err != nil { log.Fatal(err) } ``` Definitions can be gotten by Definition ID, Name or Type. We recommend using OOTB definitions and getting them by types. There is an enumerator-like helper for default types `api.RoleTypeKinds`. ### Getting principals Principal is site user or group, their IDs are used in roles assignments. Principal ID can be received by requesting UIL (User Information List), getting site user/group by name/email, etc. ```go roup, err := sp.Web().SiteGroups().GetByName("Process Users").Get() if err != nil { log.Fatal(err) } fmt.Printf("Principal ID: %d\n", group.Data().ID) // or user, err := sp.Web().SiteUsers().GetByEmail("jonh.doe@contoso.com").Get() if err != nil { log.Fatal(err) } fmt.Printf("Principal ID: %d\n", user.Data().ID) ``` It's in preference to operate on groups level and grant permissions to a specific users as little as possible. ### Adding role assignments > The role assignment is the relationship among the role definition, the users and groups, and the scope (for example, one user may be a reader on list 1, while another user is a reader on list 2). The relationship expressed through the role assignment is the key to making SharePoint security management role-based. At last, now we are ready for roles assigment. 😜 Who told permissions is simple? ```go if err := item.Roles().AddAssigment(group.Data().ID, def.ID); err != nil { log.Fatal(err) } ``` ### Removing role assignments Removing role assignments is just the same as adding but in opposite. ```go if err := item.Roles().RemoveAssigment(group.Data().ID, def.ID); err != nil { log.Fatal(err) } ``` ### Reseting roles inheritance To reset permissions inheritance: ```go item := list.Items().GetByID(1) if err := item.Roles().ResetInheritance(); err != nil { log.Fatal(err) } ``` After reseting object roles assigments its permissions are again inherited from the parent object. ### Getting collection objects' permissions When dealing with OData collection of objects with unique permissions OData items' role assigment can be requested using `RoleAssignments`, also `HasUniqueRoleAssignments` can be used in moderation. > Please be aware that `HasUniqueRoleAssignments` is a heavy property that creates workload on a SharePoint server and better be used at minimal. Try not abusing it by potential requests to large lists getting a bunch of items. ```go // Getting all lists with role assigments details res, err := sp.Web().Lists(). Select(` Title, RoleAssignments/Member/*, RoleAssignments/RoleDefinitionBindings/* `). Expand(` RoleAssignments/Member, RoleAssignments/RoleDefinitionBindings `). Get() ``` [See more](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/permissions). `RoleAssigments` if any applied contains an array of objects. ```go var lists []*struct { Title string RoleAssignments []*api.RoleAssigment } // .Normalized() method aligns responses between different OData modes if err := json.Unmarshal(res.Normalized(), &lists); err != nil { log.Fatalf("unable to parse the response: %v", err) } ``` Assigments is a Member and binded RoleDefinitions: ```go // RoleAssigment role asigments model type RoleAssigment struct { Member *struct { LoginName string PrincipalType int } RoleDefinitionBindings []*RoleDefInfo } ``` ### Base permissions `BasePermissions` is permissions representation with `Low` and `High` pair. Don't panic if API returns only BasePermissions (`{ "High": "2147483647", "Low": "4294705151" }`), using `HasPermissions` helper it's simple to check if it includes required permissions kind: ```go // User effective base permissions effectiveBasePermissions := api.BasePermissions{ High: 432, Low: 1011030767, } hasPerm := api.HasPermissions( effectiveBasePermissions, api.PermissionKind.EditListItems) ``` ### Summary Now we know how to treat SharePoint permissions using Gosip and Fluent API. Before wrapping up, we want to stress on importance of careful planning of permissions model, as less unique permissions is better. # Groups & Users Managing groups, requesting users Site groups and users can be requested via Web's `.SiteGroups()` and `.SiteUsers()` queriable collections correspondingly. ### Getting users sample ```go users, err := sp.Web().SiteUsers(). Select("Email,Id"). Filter("Email ne ''"). OrderBy("Id", true). Get() if err != nil { log.Fatal(err) } for _, user := range users.Data() { fmt.Printf("%d: %s\n", user.Data().ID, user.Data().Email) } ``` ### Getting groups sample ```go groups, err := sp.Web().SiteGroups().Get() if err != nil { log.Fatal(err) } for _, group := range groups.Data() { fmt.Printf("%d: %s\n", user.Data().ID) } ``` ### Ensuring a user You can't add new user via SharePoint API, but a user who exists in AD/AAD can be added to a site by ensuring him/her by a logon name. ```go user, err := sp.Web().EnsureUser("jane.doe@contoso.onmicrosoft.com") if err != nil { log.Fatal(err) } fmt.Printf("User: %+v\n", user) ``` ### Associated groups We love the "power of defaults". Each web by default has three predefined groups: Owners, Members and Visitors. But their IDs and names are different from web to web. Luckily there is a helper for getting associated groups. ```go // .Members() and .Owners() correspondingly group, err := sp.Web().AssociatedGroups().Visitors().Get() if err != nil { log.Fatal(err) } fmt.Printf( "Visitors group ID %d, Title \"%s\"\n", group.Data().ID, group.Data().Title, ) ``` ### Creating groups ```go group, err := sp.Web().SiteGroups().Add("My group", nil) if err != nil { log.Fatal(err) } fmt.Printf("New group ID: %d\n", group.Data().ID) ``` ### Deleting groups ```go // or .RemoveByID(groupID) if err := sp.Web().SiteGroups().RemoveByLoginName("My group"); err != nil { log.Fatal(err) } ``` ### Adding user to a group ```go user, err := sp.Web().EnsureUser("jane.doe@contoso.onmicrosoft.com") if err != nil { log.Fatal(err) } fmt.Printf("User login: %s\n", user.LoginName) // User login: i:0#.f|membership|jane.doe@contoso.onmicrosoft.com visitorGroup := sp.Web().AssociatedGroups().Visitors() if err := visitorGroup.AddUser(user.LoginName); err != nil { log.Fatal(err) } ``` In group's `.AddUser` method the argument should be full and valid login name including security provider membership prefix. For instance, while you can ensure Jane using `jane.doe@contoso.onmicrosoft.com` the same as an `.AddUser` method will fail as Jane's login is actually different `i:0#.f|membership|jane.doe@contoso.onmicrosoft.com`. If you already know UserID but not sure about LoginName `.AddUserByID` helper is at the disposal. ### Removing users from a group Similarly as with adding users: ```go user, err := sp.Web().EnsureUser("jane.doe@contoso.onmicrosoft.com") if err != nil { log.Fatal(err) } memberGroup := sp.Web().AssociatedGroups().Members() if err := memberGroup.RemoveUser(user.LoginName); err != nil { log.Fatal(err) } // or if err := memberGroup.RemoveUserByID(user.ID); err != nil { log.Fatal(err) } ``` ### Managing group owner ```go memberGroup := sp.Web().AssociatedGroups().Members() if err := memberGroup.SetAsOwner(user.ID); err != nil { log.Fatal(err) } ``` ### Getting group's users ```go users, err := sp.Web().AssociatedGroups().Visitors(). Users().Select("Id,Title").Get() if err != nil { log.Fatal(err) } for _, user := range users.Data() { fmt.Printf("%d: %s\n", user.Data().ID, user.Data().Title) } ``` ### Summary That's it, most of the common actions with groups and users are covered. You'd probably will be interested with the connected topics: {% content-ref url="/pages/-LxkLSnDGTrBn6lgttOg" %} [Permissions](/samples/permissions) {% endcontent-ref %} {% content-ref url="/pages/-LxkiGFcOQJiJC\_I87an" %} [Search API](/samples/search-api) {% endcontent-ref %} {% content-ref url="/pages/-LxkixnLbTA7k0W0meXu" %} [User Profiles](/samples/user-profiles) {% endcontent-ref %} # Search API Searching content via SharePoint API Search and search API is a huge and complex topic. In this article we're going mostly cover some basics in combination of Gosip plus REST API. ## Basic search call While executing search call there is no difference what web/site is used as a context, search API returns data due to search query object. The simplest request looks this way: ```go // `res`ult here is a byte array with some helper methods as .Results() res, err := sp.Search().PostQuery(&api.SearchQuery{ QueryText: "*", RowLimit: 5, }) if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", res.Results()) ``` ## Search query Search query struct (`api.SearchQuery`) contains some, let's say, reasonable amount of options 🙈. | Option | Description | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | QueryText | A string that contains the text for the search query | | QueryTemplate | A string that contains the text that replaces the query text, as part of a query transform | | EnableInterleaving | A Boolean value that specifies whether the result tables that are returned for the result block are mixed with the result tables that are returned for the original query | | EnableStemming | A Boolean value that specifies whether stemming is enabled | | TrimDuplicates | A Boolean value that specifies whether duplicate items are removed from the results | | EnableNicknames | A Boolean value that specifies whether the exact terms in the search query are used to find matches, or if nicknames are used also | | EnableFQL | A Boolean value that specifies whether the query uses the FAST Query Language (FQL) | | EnablePhonetic | A Boolean value that specifies whether the phonetic forms of the query terms are used to find matches | | BypassResultTypes | A Boolean value that specifies whether to perform result type processing for the query | | ProcessBestBets | A Boolean value that specifies whether to return best bet results for the query. This parameter is used only when EnableQueryRules is set to true, otherwise it is ignored. | | EnableQueryRules | A Boolean value that specifies whether to enable query rules for the query | | EnableSorting | A Boolean value that specifies whether to sort search results | | GenerateBlockRankLog | Specifies whether to return block rank log information in the BlockRankLog property of the interleaved result table. A block rank log contains the textual information on the block score and the documents that were de-duplicated. | | SourceID | The result source ID to use for executing the search query | | RankingModelID | The ID of the ranking model to use for the query | | StartRow | The first row that is included in the search results that are returned. You use this parameter when you want to implement paging for search results. | | RowLimit | The maximum number of rows overall that are returned in the search results. Compared to RowsPerPage, RowLimit is the maximum number of rows returned overall. | | RowsPerPage | The maximum number of rows to return per page. Compared to RowLimit, RowsPerPage refers to the maximum number of rows to return per page, and is used primarily when you want to implement paging for search results. | | SelectProperties | The managed properties to return in the search results | | Culture | The locale ID (LCID) for the query | | RefinementFilters | The set of refinement filters used when issuing a refinement query (FQL) | | Refiners | The set of refiners to return in a search result | | HiddenConstraints | The additional query terms to append to the query | | Timeout | The amount of time in milliseconds before the query request times out | | HitHighlightedProperties | The properties to highlight in the search result summary when the property value matches the search terms entered by the user | | ClientType | The type of the client that issued the query | | PersonalizationData | The GUID for the user who submitted the search query | | ResultsURL | The URL for the search results page | | QueryTag | Custom tags that identify the query. You can specify multiple query tags | | ProcessPersonalFavorites | A Boolean value that specifies whether to return personal favorites with the search results | | QueryTemplatePropertiesURL | The location of the queryparametertemplate.xml file. This file is used to enable anonymous users to make Search REST queries | | HitHighlightedMultivaluePropertyLimit | The number of properties to show hit highlighting for in the search results | | EnableOrderingHitHighlightedProperty | A Boolean value that specifies whether the hit highlighted properties can be ordered | | CollapseSpecification | The managed properties that are used to determine how to collapse individual search results. Results are collapsed into one or a specified number of results if they match any of the individual collapse specifications. In a collapse specification, results are collapsed if their properties match all individual properties in the collapse specification. | | UIlanguage | The locale identifier (LCID) of the user interface | | DesiredSnippetLength | The preferred number of characters to display in the hit-highlighted summary generated for a search result | | MaxSnippetLength | The maximum number of characters to display in the hit-highlighted summary generated for a search result | | SummaryLength | The number of characters to display in the result summary for a search result | | SortList | The list of properties by which the search results are ordered | | Properties | Properties to be used to configure the search query | | ReorderingRules | Special rules for reordering search results. These rules can specify that documents matching certain conditions are ranked higher or lower in the results. This property applies only when search results are sorted based on rank. | ![](https://github.githubassets.com/images/icons/emoji/trollface.png) But don't be afraid you use only a few of them and on demand and when know what you need. ### User profiles search sample ```go res, err := sp.Search().PostQuery(&api.SearchQuery{ QueryText: "*", RowLimit: 5, SelectProperties: []string{"AccountName", "Title", "Department", "JobTitle"}, StartRow: 10, TrimDuplicates: false, SourceID: "b09a7990-05ea-4af9-81ef-edfab16c4e31", SortList: []*api.SearchSort{ &api.SearchSort{ Property: "Title", Direction: 0, }, }, }) if err != nil { log.Fatal(err) } fmt.Printf("AccountName | Title | Department | JobTitle\n") fmt.Printf("------------|-------|------------|---------\n") for _, profile := range res.Results() { fmt.Printf( "%s | %s | %s | %s\n", profile["AccountName"], profile["Title"], profile["Department"], profile["JobTitle"], ) } ``` This search request searched only for user profiles (defined by SourceID), return five results per page, skips first ten records ignores trimming duplicates, retrieves specific managed properties and sorts by Title property. ### Search response Search response in Fluent API is sligtly bit adjusted with helper methods. Result itself is byte array, so you can process it in a custom way. For lazy people (as I am) there are `.Data()` and `.Results()` helpers. #### Results helper Results helper retrieves `PrimaryQueryResult.RelevantResults.Table.Rows` and reduces search response results to the convenient array of string maps. So you can grasp results this way: ```go for _, profile := range res.Results() { fmt.Printf( "%s | %s | %s | %s\n", profile["AccountName"], profile["Title"], profile["Department"], profile["JobTitle"], ) } ``` #### Data helper Data helper provides more options. It returns the following struct: ```go type SearchResults struct { ElapsedTime int PrimaryQueryResult *ResultTableCollection Properties []*TypedKeyValue SecondaryQueryResults []*ResultTableCollection SpellingSuggestion string TriggeredRules []interface{} } type ResultTableCollection struct { QueryErrors map[string]interface{} QueryID string QueryRuleID string CustomResults *ResultTable RefinementResults *ResultTable RelevantResults *ResultTable SpecialTermResults *ResultTable } ``` So you can have access to refiners and many more search goodies. # User Profiles Dealing with user profiles API Working with UPS API is simple. UPS API is not something sophisticated as [Search](https://app.gitbook.com/@koltyakov/s/gosip/~/drafts/-LxpsgqFJPaz-EgS5dYZ/samples/search-api) for example and it has just a few methods which can be in demand in a server-side operation. Don't get me wrong, there are not-covered social features from within user profiles API in Gosip but we rarely have seen demand and scenarios of their usage in SharePoint solutions. Anyways, if some additional feature coverage is required for you please let us know by posting an [issue](https://github.com/koltyakov/gosip/issues). ### Getting profiles The most in-demand, I would say, feature when it comes to user profiles is getting all profiles. However, UPS API allows only dealing with a single profile, you can't request all of them. Luckily, this is possible and recommended achieving with the search. ```go res, err := sp.Search().PostQuery(&api.SearchQuery{ QueryText: "*", SourceID: "b09a7990-05ea-4af9-81ef-edfab16c4e31", }) if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", res.Results()) ``` See a bit more [advanced sample](/samples/search-api#user-profiles-search-sample). ### Getting profile properties ```go user, err := sp.Web().SiteUsers(). GetByEmail("jane.doe@contoso.onmicrosoft.com").Get() if err != nil { log.Fatal(err) } props, err := sp.Profiles().GetPropertiesFor(user.Data().LoginName) if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", props.Data()) ``` Gosip strongly types properties response using `.Data()` helper, here is the resulted struct: ```go type ProfilePropsInto struct { AccountName string DirectReports []string DisplayName string Email string ExtendedManagers []string ExtendedReports []string Peers []string IsFollowed bool PersonalSiteHostURL string PersonalURL string PictureURL string Title string UserURL string UserProfileProperties []*TypedKeyValue } ``` ### Getting single property Sometimes you have to be as effective as possible and trim down responses to a minimum. Let's say you only need a single property. ```go rop, err := sp.Profiles(). GetUserProfilePropertyFor(user.Data().LoginName, "AccountName") if err != nil { log.Fatal(err) } fmt.Printf("%s\n", prop) ``` Seriously, I don't know the reason for existing of getting a single property, but not getting specific properties or multiple profiles or updating multiple properties. There are no excuses yet this part of SharePoint API is clunky and really old. ### Setting user profile property values There are two methods for setting user profile property value. Yeah, you heard me the right property in a time. #### Set single value profile property ```go if err := sp.Profiles().SetSingleValueProfileProperty( user.Data().LoginName, "AboutMe", "Updated from Gosip", ); err != nil { log.Fatal(err) } ``` #### Set multi valued profile property ```go tags := []string{"#ci", "#demo", "#test"} if err := sp.Profiles().SetMultiValuedProfileProperty( user.Data().LoginName, "SPS-HashTags", tags, ); err != nil { log.Fatal(err) } ``` ### Summary User profiles API is limited due to its legacy nature. It is what it is. However, many SharePoint solutions, especially intranet portals and workflow processes can be heavily based on UPS. Go worker can be handy for custom synchronization scenarios and also in external workflow workers with UPS as a source for settings for detecting user dynamic roles. # Change API Getting changes, synchronisation scenarios When it comes to synchronization or delta processing Change API is for the rescue. There is no need in any sort of logic that used modified date comparison to detect what has been changed since the last processing time. There is no need to wait until the search crawl is done to check changes through web or site collection. And not only these benefits but also tracking permissions changes restores from recycle bin, system updates and much more. Changes API is what is used together with Webhooks (in SPO), by having a context (scope) and change token(s) you can easily retrieve what has been changed and run the custom logic based on the nature of changes. Changes can be requested from within different scopes: lists, webs, sites, etc. Change API operates change tokens to establish start and end anchors. Change tokens received from a specific entity type (e.g. Site) can't be used with another entity type (e.g. List) while sending change query. ### Getting current change token ```go siteChangeToken, err := sp.Site().Changes().GetCurrentToken() if err != nil { log.Fatal(err) } fmt.Printf("Site change token: %s\n", siteChangeToken) webChangeToken, err := sp.Web().Changes().GetCurrentToken() if err != nil { log.Fatal(err) } fmt.Printf("Web change token: %s\n", webChangeToken) list := sp.Web().GetList("Lists/MyList") listChangeToken, err := list.Changes().GetCurrentToken() if err != nil { log.Fatal(err) } fmt.Printf("List change token: %s\n", listChangeToken) // Site change token: // 1;1;afbc6f5a-65b3-4b64-9f40-885d8d772c8c;637138201430100000;64696075 // Web change token: // 1;2;7e084f68-c401-4910-b75e-4347c7848965;637138201430100000;64696075 // List change token: // 1;3;3b542696-5863-4ff7-bb90-8224e9e8adb9;637138201430100000;64696075 ``` ### Getting changes Knowing what root entities' token(s) you have the specific changes query can be crafted and sent, e.g.: ```go list := sp.Web().GetList("Lists/MyList") listChangeToken, _ := list.Changes().GetCurrentToken() list.Items().Add([]byte(`{"Title":"New item"}`)) // error handling is omitted -- adding a dummy item to receive changes result changes, err := list.Changes().GetChanges(&api.ChangeQuery{ ChangeTokenStart: listChangeToken, List: true, Item: true, Add: true, }) if err != nil { log.Fatal(err) } for _, change := range changes.Data() { fmt.Printf("%+v\n", change) } ``` Change results are strongly typed with Gosip's API, the `changes` variable in the sample is an array of `api.ChangeInfo` struct pointers. Change into struct contains the following properties: ```go type ChangeInfo struct { ChangeToken *StringValue ChangeType int Editor string EditorEmailHint string ItemID int ListID string ServerRelativeURL string SharedByUser string SharedWithUsers string SiteID string Time time.Time UniqueID string WebID string } ``` `ChangeType` is an important piece of information, it's an enumerator describing a nature of a change. See [more](https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-csom/ee543793%28v%3Doffice.15%29). {% hint style="info" %} Change API doesn't return what specifically was changed but only where it was changed. For instance, when an item is changed the API will return its identities information but not the metadata. {% endhint %} In a synchronization scenario or a Webhook after getting information which items are changed the corresponding request(s) should be sent for getting specifics. ### Change query When getting the changes the API requires some clarifications about what exactly should be retrieved. This is defined within a change query which is implemented `api.ChangeQuery` struct. | Property | Description | | --------------------- | ---------------------------------------------------------------------------------------------------- | | ChangeTokenStart | Specifies the start date and start time for changes that are returned through the query | | ChangeTokenEnd | Specifies the end date and end time for changes that are returned through the query | | Add | Specifies whether add changes are included in the query | | Alert | Specifies whether changes to alerts are included in the query | | ContentType | Specifies whether changes to content types are included in the query | | DeleteObject | Specifies whether deleted objects are included in the query | | Field | Specifies whether changes to fields are included in the query | | File | Specifies whether changes to files are included in the query | | Folder | Specifies whether changes to folders are included in the query | | Group | Specifies whether changes to groups are included in the query | | GroupMembershipAdd | Specifies whether adding users to groups is included in the query | | GroupMembershipDelete | Specifies whether deleting users from the groups is included in the query | | Item | Specifies whether general changes to list items are included in the query | | List | Specifies whether changes to lists are included in the query | | Move | Specifies whether move changes are included in the query | | Navigation | Specifies whether changes to the navigation structure of a site collection are included in the query | | Rename | Specifies whether renaming changes are included in the query | | Restore | Specifies whether restoring items from the recycle bin or from backups is included in the query | | RoleAssignmentAdd | Specifies whether adding role assignments is included in the query | | RoleAssignmentDelete | Specifies whether adding role assignments is included in the query | | RoleDefinitionAdd | Specifies whether adding role assignments is included in the query | | RoleDefinitionDelete | Specifies whether adding role assignments is included in the query | | RoleDefinitionUpdate | Specifies whether adding role assignments is included in the query | | SecurityPolicy | Specifies whether modifications to security policies are included in the query | | Site | Specifies whether changes to site collections are included in the query | | SystemUpdate | Specifies whether updates made using the item SystemUpdate method are included in the query | | Update | Specifies whether update changes are included in the query | | User | Specifies whether changes to users are included in the query | | View | Specifies whether changes to views are included in the query | | Web | Specifies whether changes to Web sites are included in the query | ### Pagination Sometimes it can be lots of changes since a provided token. Gosip implements pagination helper using "jumping" between different change tokens under the hood. When last change item's token is used as a start token to get the "next page". This approach is used in `GetNextPage`: ```go changesFirstPage, _ := list.Changes().Top(100).GetChanges(&ChangeQuery{ ChangeTokenStart: token, List: true, Item: true, Add: true, }) changesSecondPage, _ := changesFirstPage.GetNextPage() ``` ### Summary Change API is a powerful mechanism and a robust way for processing delta changes that can and should be used in advanced and optimised synchronizations and business processes withing external workers. # Attachments Dealing with items attachments Our common recommendation with attachments is to use them in moderation with a preference to documents in libraries and linking business objects items to that document using metadata and other logical relationship. But sometimes you need nothing more than just a simple binary addition to an item. Working with attachments is mostly straightforward as you can only get a list of item's attachment, get a specific attachment by its name, add and delete an attachment. ### Getting attachments ```go list := sp.Web().GetList("Lists/MyList") item := list.Items().GetByID(1) attachments, err := item.Attachments().Get() if err != nil { log.Fatal(err) } for _, attachment := range attachments.Data() { data := attachment.Data() fmt.Printf("%s (%s)\n", data.FileName, data.ServerRelativeURL) } ``` Attachments API provides little information, actually only FileName and ServerRelativeURL. ### Items have attachments To detect which items have attachments the corresponding `Attachments` property can be requested within an ordinary get items request: ```go items, err := list.Items().Select("Id,Attachments").Get() if err != nil { log.Fatal(err) } for _, item := range items.Data() { data := item.Data() hasAttachments := "has no" if data.Attachments { hasAttachments = "has" } fmt.Printf("Item ID %d %s attachments\n", data.ID, hasAttachments) } ``` ### Adding attachments Adding attachments is almost identical to [adding documents](/samples/documents#adding-uploading-a-file) to a document library. ```go list := sp.Web().GetList("Lists/MyList") item := list.Items().GetByID(1) content := strings.NewReader("Get content in a usual Go way you like") item.Attachments().GetByName("MyAttachment.txt").Delete() resp, err := item.Attachments().Add("MyAttachment.txt", content) if err != nil { log.Fatal(err) } fmt.Printf("Attachment is added: %s\n", resp.Data().ServerRelativeURL) ``` ### Getting attachment by name ```go list := sp.Web().GetList("Lists/MyList") item := list.Items().GetByID(1) content, err := item.Attachments().GetByName("MyAttachment.txt").Download() if err != nil { log.Fatal(err) } fmt.Printf( "Do whatever needed with this content of %d bytes\n", len(content), ) ``` ### Attachment actions With an attachment you can: * Download * Get reader (download in a stream way) * Delete * or Recycle These actions are rather obvious with the help of the Fluent API. # Record Management In place record management helpers In enterprise content management (ECM) scenarios, records are an important element within a document lifecycle and a company's compliance policies. Not only documents but actually items. When declared as a record a document or item can't be later changed or moved without undeclaring. This guarantees integrity. REST API in terms of managing classic in-place records is limited, you can only retrieve record status from item metadata. Item's `OData__vti_ItemDeclaredRecord` property tells was an item declared as a record and when. The field is a date-time value, empty means item is not a record. We would not be we if didn't add an extension with allows Gosip to declare, undeclare and declare with a declaration date. To workaround REST's limitation, these functionality is achieved by corresponding CSOM calls. What's great is that Gosip abstracts for you such a nuance providing methods which under-the-hood details are interesting but not necessarily should be known by a library consumer. It's important to notice that in-place record management requires some [configuration](https://support.office.com/en-us/article/configuring-in-place-records-management-7707a878-780c-4be6-9cb0-9718ecde050a) which is not the topic of this article. So let's observe API methods instead. ### Is item declared as record ```go list := sp.Web().GetList("Lists/MyList") item := list.Items().GetByID(1) isRecord, err := item.Records().IsRecord() if err != nil { log.Fatal(err) } fmt.Printf("Item is record? %t\n", isRecord) ``` ### Is document declared as a record Declaration as records is more common for documents (files), yet it's actually a file's item that keeps record status. And therefore holds API actions. ```go file := sp.Web().GetFile("MyLibrary/Contract.docx") item, err := file.GetItem() if err != nil { log.Fatal(err) } isRecord, err := item.Records().IsRecord() if err != nil { log.Fatal(err) } fmt.Printf("Document is record? %t\n", isRecord) ``` ### Declare as a record ```go if err := item.Records().Declare(); err != nil { log.Fatal(err) } ``` ### Undeclare as a record ```go if err := item.Records().Undeclare(); err != nil { log.Fatal(err) } ``` ### Declare with declaration date {% hint style="warning" %} The support of that method is not presented in old SharePoint versions. {% endhint %} ```go date, _ := time.Parse(time.RFC3339, "2019-01-01T08:00:00.000Z") if err := item.Records().DeclareWithDate(date); err != nil { log.Fatal(err) } ``` ### Summary There is no methods for declaring and undeclaring items as records in SharePoint REST API (yet) and there won't be ever, IMO, for classic record management, but maybe to the [modern](https://regarding365.com/modern-vs-classic-in-place-records-management-in-sharepoint-8424ebab29f1), however, in Gosip, we mimic some CSOM calls and providing corresponding methods. # Sending Emails Email notifications utility There is nothing simpler than sending email notifications using SharePoint REST and Gosip Fluent API. However, when building workflow workers or custom subscription services email functionality is vital. And what can be better than embedded OOTB functionality? No extra knowledge of SMTP server and mail credentials, just a usual API call to `_api/SP.Utilities.Utility.SendEmail` utility endpoint. There are some limitations to this method: * recipients only from the site * impossibility to change the sender * no attachments but as embedded solution it's OK. And probably sort of workflow notification service should stay in such margins anyways. ### Sending email notification ```go user, err := sp.Web().SiteUsers(). GetByEmail("jane.doe@contoso.onmicrosoft.com").Get() if err != nil { log.Fatal(err) } if err := sp.Utility().SendEmail(&api.EmailProps{ Subject: "Say hi to Gosip!", Body: "Text or HTML body here...", To: []string{user.Data().Email}, }); err != nil { log.Fatal(err) } ``` Send email utility is not sophisticated but it works and enough for an embedded functionality. # Property Bags Property bags operations Property bags are a nice way of storing global metadata and settings in SharePoint. Property bags are key-value pairs scoped to the container. In general, any folder can act as a container, also webs have their own property bag storage located in all properties section. Use-cases can vary depending on how an application uses this key-value storage. For example, PnP Provisioning engine stores applied schema information in property bags. A good thing is that you don't need to provision any additional artifacts to keep some business logic or state variables. A thing to know that all the props values are strings, dealing with different data types they should be serialized or converted to a string. ### Getting web's all properties ```go props, err := sp.Web().AllProps().Get() if err != nil { log.Fatal(err) } for key, val := range props.Data() { fmt.Printf("%s: %s\n", key, val) } ``` ### Getting specific properties To get a limited subset of properties `.GetProps` helper method can be used: ```go props, err := sp.Web().AllProps().GetProps([]string{ "taxonomyhiddenlist", "vti_defaultlanguage", }) if err != nil { log.Fatal(err) } for key, val := range props { fmt.Printf("%s: %s\n", key, val) } ``` Selecting specific properties didn't work in old versions of SharePoint, however, the method has a fallback to getting all and filtering props on the client. ### Setting property value ```go if err := sp.Web().AllProps().Set("my-prop", "my value"); err != nil { log.Fatal(err) } ``` Modern SharePoint sites by default have custom scripting disabled mode. When custom scripting is disabled even an admin account will receive "Access denied. You do not have permission to perform this action or access this resource." error message. This is the expected behavior. {% hint style="warning" %} Setting props require "Custom Script" be allowed on a site. See [more](https://docs.microsoft.com/en-us/sharepoint/allow-or-prevent-custom-script). {% endhint %} {% hint style="success" %} We recommend [Office 365 CLI](https://pnp.github.io/office365-cli/) to enable custom scripting. {% endhint %} ```bash spo site classic set --url https://contoso/sites/site --noScriptSite false ``` ### Getting folder property bags ```go props, err := sp.Web().GetFolder("MyLibrary").Props().Get() if err != nil { log.Fatal(err) } for key, val := range props.Data() { fmt.Printf("%s: %s\n", key, val) } ``` ### Setting many properties values ```go err := sp.Web().GetFolder("MyLibrary").Props(). SetProps(map[string]string{ "prop01": "value 01", "prop02": "value 02", }) if err != nil { log.Fatal(err) } ``` ### Summary Property bags are a robust way of storing custom settings and state which requires no additional artifacts. When structuring and consuming correctly they can be a great addition to the application logic. # Recycle Bin Recycling methods and dealing with recycle bin You can work with recycle bins via REST API similarly as with lists. ### Getting deleted items ```go data, err := sp.Site().RecycleBin(). OrderBy("DeletedDate", false). Top(5). Get() // site's Recycle Bin // data, err := sp.Web().RecycleBin().Get() // web's one if err != nil { log.Fatal(err) } for _, item := range data.Data() { d := item.Data() fmt.Println( d.ID, d.ItemType, d.LeafNamePath.DecodedURL, d.DeletedByName, d.DeletedDate, ) } ``` Items in recycle bins are queryable collection, OData modifiers can be applied in a usual way. Response is strongly typed, helps do not care about unmarshalling the structures. Items in recycle bin contains the following metadata: ```go type RecycledItem struct { AuthorEmail string AuthorName string DeletedByEmail string DeletedByName string DeletedDate time.Time DeletedDateLocalFormatted string DirName string ID string ItemState int ItemType int LeafName string Size int Title string LeafNamePath *DecodedURL DirNamePath *DecodedURL } ``` Once you have Item ID (which is a GUID in case of recycle bin) you can not resore it. ### Restoring recycled items ```go data, err := sp.Site().RecycleBin().Top(1).Get() if err != nil { log.Fatal(err) } if len(data.Data()) > 0 { itemID := data.Data()[0].Data().ID if err := sp.Site().RecycleBin().GetByID(itemID).Restore(); err != nil { log.Fatal(err) } } ``` # Feature management Operations for features management You can activate and deactivate features on sites and webs using REST API. However, there are a few nuances to know knowing which makes it simple to manage the features. {% hint style="info" %} You should know specific feature definition ID to add or remove it. {% endhint %} Unfortunately, there is no way of getting features list with names and descriptions programmatically. When getting features list on the web or site you receive a list of activated features definition IDs and nothing else. ### Getting activated features ```go res, err := sp.Web().Features().Get() if err != nil { log.Fatal(err) } for _, f := range res { fmt.Printf("%+v\n", f) } // &{DefinitionID:b77b6484-364e-4356-8c72-1bb55b81c6b3} // &{DefinitionID:a7a2793e-67cd-4dc1-9fd0-43f61581207a} // &{DefinitionID:d5a4ed08-27b9-4142-9804-45dec6fda126} // &{DefinitionID:780ac353-eaf8-4ac2-8c47-536d93c03fd6} // &{DefinitionID:8c6f9096-388d-4eed-96ff-698b3ec46fc4} // ... ``` Luckily, nowadays one almost always deal only with OOTB features, if not probably something is terribly wrong on a project. 😝 ### Feature activating ```go mds := "87294c72-f260-42f3-a41b-981a2ffce37a" if err := sp.Web().Features().Add(mds, true); err != nil { log.Fatal(err) } ``` The first argument stands for Feature Definition ID, the second one is force mode. ### Feature deactivating ```go mds := "87294c72-f260-42f3-a41b-981a2ffce37a" if err := sp.Web().Features().Remove(mds, true); err != nil { log.Fatal(err) } ``` Arguments are identical. When a feature was not activated before removing it an error message is expected ("Feature is not activated at this scope"). Site level actions are absolutely identical, the only difference is `sp.Site()` entry point. # Advanced item requests Advanced scenarios for getting list items There are so many nuances connected with requesting items in a list. By default, it's recommended using OData operations as the most simple, straightforward and RESTful approach. {% content-ref url="/pages/-Lxfmyh0dc-ZpwVgL3To" %} [Basic CRUD](/samples/basic-crud) {% endcontent-ref %} But OData doesn't cover all of the needs and you got to switch back but not necessarily backward to [CAML](https://docs.microsoft.com/en-us/sharepoint/dev/schema/query-schema) methods. There a lot of gaps which can force you doing this: single valued MMD fields, group by requests, getting items from a view, to name just a few. Sometimes it's old known bugs or limitations, sometimes a specific functionality which was in the CAML days. ### Get items by CAML ```go list := sp.Web().GetList("Lists/MyList") caml := ` 3 ` data, err := list.Items().GetByCAML(caml) if err != nil { log.Fatal(err) } fmt.Printf("%s\n", data) ``` Of course, CAML query can be slightly bit more complex than just this. 😁 For example, you can request data as in a list view: ```go list := sp.Web().GetList("Lists/MyList") viewResp, err := list.Views().DefaultView().Get() if err != nil { log.Fatal(err) } data, err := list.Items().GetByCAML(viewResp.Data().ListViewXML) if err != nil { log.Fatal(err) } fmt.Printf("%s\n", data) ``` However, when working with CAML there is more powerful methods. ### Render List Data ```go list := sp.Web().GetList("Lists/MyList") caml := ` ` data, err := list.RenderListData(caml) if err != nil { log.Fatal(err) } fmt.Printf("%s\n", data) ``` RenderListData\* methods have completely different response structure. It's good and bad at the same time. The cons are that the results are not compatible and really different from OData methods, not only the shape but also values format. Sometimes it aches. The pros are that render list data methods provide something which is missed in OData responses: group by with collapsed data and only subtotals are possible, recurrent calendar events can be requested (yet it's now so shiny), to name a few. RenderListData deals with GET request, which is a disadvantage as CAML query length is limited. This limitation is rarely can be met in practice, but the ridiculously complex conditions could fail due to this fact. ### Render List Data as Stream This method is super powerful, SharePoint Modern UI list views work using `.RenderListDataAsStream`. The method is the continuation of evolving of `.RenderListData` enhanced by the vendor for their practical needs of building the modern UI views. The method deals with POST requests, almost have no length limitation, and operates with many-many options for covering all those aspects and features of the Modern UI view. {% hint style="info" %} The method is in our plans to implement in Gosip Fluent API. But you always can craft that API consumption using AdHoc queries and [HTTP Client](/api/http-client#post). {% endhint %} ### Pagination Pagination in SharePoint lists is painful if you misses couple of moments. In a contrast to many databases queries where `top` and `skip` are straightforward thing to build an pagination, OData's `$skiptoken` is not what many think. First of all, it's not a number of rows to skip before starting returning items on a next paged collection. The simplest format of skip token is: `Paged=TRUE&p_ID=5` The simplest reverse skip token is: `Paged=TRUE&PagedPrev=TRUE&p_ID=5` {% hint style="info" %} Reverse token returns previous page content obviously. {% endhint %} By looking at `p_ID` part you'd think that item's ID is enough to construct a correct skip token, but it's not so. It's only correct for not sorted collections. If any `$orderby` modifier is applied, the amount of `p_*` parameters changes. Let's assume you sorted the list by Title, it will add something like `p_Title=Smth`, where "Smth" is the last row's Title value on current page collection. Fortunately, skip tokens should not be constructed manually. REST return next page collection URI together with the responses for the current page collection. In Gosip we have helper methods which makes it simpler working with pagination. ```go list := sp.Web().GetList("Lists/MyList") page, err := list.Items().Select("Id").Top(100).GetPaged() if err != nil { log.Fatal(err) } fmt.Printf("Page items %d\n", len(page.Items.Data())) if page.HasNextPage() { nextPage, err := page.GetNextPage() if err != nil { log.Fatal(err) } fmt.Printf("Next page items %d\n", len(nextPage.Items.Data())) } ``` {% hint style="warning" %} We're planning some improvements with pagination interfaces and scale the approach to all possible paged collections APIs but will try to introduce any backward incompatibilities as little as possible. {% endhint %} ### Requesting large lists Large lists in SharePoint are those which amount of items is larger than view throttling limitation, the default limitation is 5000 items in a view. In On-Prem this value can be tweaked, in SPO this is a hard limit. REST API can't return more than 5K items at once. Filter conditions based on indexing fields must be applied to trim down items number. It can be hard, though it's common for server-side processing requesting all items even if there are tens of thousands of items in a list. Such operations are not for immidiate actions but long running syncronizations. In Gosip, `.GetAll` method is at disposal. ```go list := sp.Web().GetList("Lists/MyList") allItems, err := list.Items().Select("Id,Title").Top(5000).GetAll() if err != nil { log.Fatal(err) } // process allItems data ``` The method disables any ordering and filtering if applied, as ordering and filtering are not compatible with large lists. Recommendations for getting all items from a large list: * Always specify only really required fields to retrieve in `Select` * Use `Top` equal to 5000 (as the default Top=100) * Consider event-based synchronizations and partial [getting changes](/samples/change-api) * Consider [search-based](/samples/search-api) logic # Advanced add/update Advanced creating and updating items In addition to standard OData [add](/samples/basic-crud#adding-an-item) and [update](/samples/basic-crud#updating-an-item) items operations REST provides such useful methods as `AddValidateUsingPath` and `ValidateUpdateListItem`. The first is only presented in modern SharePoint, it not only allows adding items right in a sub folder but also operate with form data payloads and control check in process. `ValidateUpdateListItem` is handy for operations requiring [system-like-update](https://www.linkedin.com/pulse/list-items-system-update-options-sharepoint-online-andrew-koltyakov/) logic via pure REST. In Gosip, `AddValidateUsingPath` and `ValidateUpdateListItem` are represented with `Items().AddValidate()` and `Item.UpdateValidate()` methods correspondingly: ### Add validate ```go list := sp.Web().GetList("Lists/MyList") // Method options options := &api.ValidateAddOptions{ NewDocumentUpdate: true, CheckInComment: "test", DecodedPath: "Lists/MyList/subfolder" // is optional } // Form data payload data := map[string]string{ "Title": "New item", } if _, err := list.Items().AddValidate(data, options); err != nil { log.Fatal(err) } ``` As `DecodedPath` option the relative path to folder can be provided. It's optional. The path should be relative to a web without trailing slash in the beginning. Gosip adds web relative URL automatically. `ValidateAddOptions` are also optional, when no new document update or check-in comment or folder path are ever required, a `nil` value should be passed. ```go list := sp.Web().GetList("Lists/MyList") // Form data payload data := map[string]string{ "Title": "New item", } if _, err := list.Items().AddValidate(data, nil); err != nil { log.Fatal(err) } ``` ### Update validate Using update validate is almost the same: ```go options := &ValidateUpdateOptions{ NewDocumentUpdate: true, CheckInComment: "test", } data := map[string]string{ "Title": "New item", } if _, err := list.Items().GetByID(3).UpdateValidate(data, options); err != nil { log.Fatal(err) } ``` ### Form values fingerprints Form values passed to the methods should stand for an array of `{ FieldName: "", FieldValue: "" }` objects where field value is a string of specific format depending on field's data type. Gosip simplifies this payload operating with map of strings. In payload, map key should stand for a valid `FieldName`, a value, obviously, is the one mapped to `FieldValue`. The fingerprints for the data types are following: | Field data type | Value sample | Comment | | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | Text (single line and note) | "text" | | | Number | "123" | as a string | | Yes/No | "1" | "1" - Yes, "2" - No | | Person or group, single and multiple | \`\[{ "Key": "LoginName", "IsResolved": true }]\` |

"LoginName" is a valid login name, including provider prefix

"IsResolved" is optional

| | Date time | "6/23/2018 10:15 PM" | for different web locales is different | | Date only | "6/23/2018' | for different web locales is different | | Choice (single) | "Choice 1" | | | Choice (multi) | "Choice 1;#Choice 2" | ";#" separated list | | Hyperlink or picture | ", Gosip" | a description can go after URL and ", " delimiter | | Lookup (single) | "2" | item ID as string | | Lookup (multi) | "1;#;#2;#;#3;#" | ";#" separated list, after each ID goes additional ";#" | | Managed metadata (single) | "Department 2\|220a3627-4cd3-453d-ac54-34e71483bb8a;" | | | Managed metadata (multi) | "Department 2\|220a3627-4cd3-453d-ac54-34e71483bb8a;Department 3\|700a1bc3-3ef6-41ba-8a10-d3054f58db4b;" | | # Unmarshaling responses Parsing complex responses Gosip Fluent API tries providing strongly typed responses objects, but in most situations, it's not possible as there are many factors reshaping a response JSON body. By defining different OData modes (Verbose, Minimalmetadata, Nometadata) within the Accept headers SharePoint REST API returns not only data of different details but also data in different forms payload shape-terms. Which can lead to runtime errors. We're are not forcing to use one specific OData mode only (e.g. Minimalmetadata) as this would prevent support of some old SharePoint versions (2013) without additional server-side configurations, we have responses normalisation methods which reshape JSONs from the API to the close or identical form no matter the OData mode is. Let's take a look at the example: ```go package main import ( "encoding/json" "fmt" "log" "github.com/koltyakov/gosip" "github.com/koltyakov/gosip/api" strategy "github.com/koltyakov/gosip/auth/saml" ) func main() { // Binding auth & API client configPath := "./config/private.saml.json" authCnfg := &strategy.AuthCnfg{} if err := authCnfg.ReadConfig(configPath); err != nil { log.Fatalf("unable to get config: %v", err) } client := &gosip.SPClient{AuthCnfg: authCnfg} sp := api.NewSP(client) // Getting items from a custom list list := sp.Web().GetList("Lists/MyList") data, err := list.Items().Select("Id,CustomField").Get() if err != nil { log.Fatalln(err) } // Define a stuct or map[string]interface{} for unmarshalling items := []*struct { ID int `json:"Id"` Custom string `json:"CustomField"` }{} // .Normalized() method aligns responses between different OData modes if err := json.Unmarshal(data.Normalized(), &items); err != nil { log.Fatalf("unable to parse the response: %v", err) } for _, item := range items { fmt.Printf("%+v\n", item) } } ``` [*Open on GitHub*](https://github.com/koltyakov/gosip-sandbox/blob/master/samples/unmarshaling/main.go) Almost any API byte array response has extension methods, such as `.Data()` and `.Normalize()`. The Data method uses predefined generic API entity items unmarshaling, it can be useful for the OOTB entities, e.g. Webs, Groups, Lists, to name just a few. But won't allow getting custom item values for example. When it comes to custom responses, you have to unmarshal by your own, however, Gosip also helps with this by providing normalization methods. Let's compare responses in a bit much details: ```javascript // "Accepts": "application/json;odata=verbose" { "d": { "results": [ { "Id": 134, "CustomField": "CustomValue", "ID": 134, // ... } ] } } // "Accepts": "application/json;odata=minimalmetadata" { "value": [ { "Id": 134, "CustomField": "CustomValue", "ID": 134, // ... } ], // ... } // "Accepts": "application/json;odata=nometadata" {"value":[{"Id":134,"CustomField":"CustomValue","ID":134}]} ``` As can be seen, there is a great difference in Verbose and Minimal/No metadata modes in shape-terms. The difference is mode dramatic when complex data fields come into play. After normalization, responses shape is reduced to the following: ```go [ { "CustomField": "CustomValue", "ID": 134, "Id": 134, // ... } ] ``` This makes unmarshalling in times simpler and reduce potential errors after deciding changing OData mode globally. # Overview Gosip sandbox area: samples, experiments & suggestions Have a use case but no idea how to implement? Ask in [issues section](https://github.com/koltyakov/gosip-sandbox/issues) and we'll reach you with suggestions or sample to start with. Have a noticeable example to share with the community? Reach us with a contribution suggestion. PRs are welcome! Have no particular code to share but a description of how you use the library or a blog post? Please let us know, we're happy to post a reference. ### Samples list | Sample | Description | | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Dynamic authentication](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/dynauth) | Shows a simple way of importing potentially demanded strategies and selecting one in runtime based on logic, CLI flags in the case of the sample. | | [Consumption with PowerShell](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/posh) | Shows how to retrieve auth tokens to use in a PowerShell script (is relevant for Edge auth scenarios, e.g. SharePoint behind WAP perimeter). | | [Dev API Proxy](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/proxy) | Shows how to expose SharePoint API as an anonymous endpoint for a dev server. | | [Files upload (sync)](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/sync) | Is a syncronization sample which provides a single time assets upload or/and local file system watch mode. Provides in times faster upload when any other known alternative. | | [Unmarshaling API responses](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/unmarshaling) | Basic unmarhsaling (response parsing) sample. | | [Objects Permissions](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/permissions) | The sample shows how to get permissions for OData collections. | | [Tenant API](https://github.com/koltyakov/gosip-sandbox/blob/master/samples/tenant/main.go) | Accessing SPO tenant scope API basic sample. E.g. for creating classic sites. | | [Modern Sites Management](https://github.com/koltyakov/gosip-sandbox/tree/master/samples/modernSites) | The sample shows how to create, delete Modern Sites and how to check a site's provisioning status. | # Headers presets OData modes headers presets REST API uses OData modes for controlling response verbosity. By defining different OData modes (Verbose, Minimalmetadata, Nometadata) within the Accept headers SharePoint REST API returns not only data of different details but also data in different forms payload shape-terms. Which can lead to runtime errors. {% content-ref url="/pages/-Ly-S6eJxo13WXdF39g1" %} [Unmarshaling responses](/samples/unmarshaling-responses) {% endcontent-ref %} When dealing with different versions of SharePoint there are few gotchas to remember. In old SharePoint 2013, only `Verbose` mode was allowed by default. This can be amended by installing WCF OData extensions and enabling JSON Light support, however, in our practice, a rare farm admin considering such an update. So it's better stick with a Verbose when it comes to SharePoint 2013. With SharePoint 2016 and newer, and, obviously, SharePoint Online, we'd recommend `Minimalmetadata` as a default. So the payloads could be a bit more smaller in size and effective overall. `Nometadata` mode could be tricky, from one hand it's close to `Minimalmetadata` yet doesn't content some vital information such as entity identities and paged collections helpers. We prefer `Minimalmetadata` over `Nometadata` in general. Gosip provides some presets for headers which could be handy together with `Conf` method. ```go var client *gosip.SPClient // ... sp := api.NewSP(client).Conf(api.HeadersPresets.Minimalmetadata) // api.HeadersPresets.Verbose // api.HeadersPresets.Nometadata ``` Along with OData modes, these presets define language header `"Accept-Language": "en-US,en;q=0.9"` which forces English messages in responses if English is installed on a site. Which is handy for dev and debugging purposes as sometimes a local non-latin language can be escaped to an unreadable form making it uncomfortable detecting what was wrong in logs. # Cpass 🔐 Simple secure string password convertor Cpass is simplified secured password two-ways encryption sub package [port](https://github.com/koltyakov/cpass) for Gosip. By default, Cpass uses Machine ID as an encryption key so a secret hash can only be decrypted on a machine where it was generated. Cpass's approach is appropriate in local development scenarios. The main goal is "not to show raw secret while presenting a desktop" or "not to commit raw secret by an incident to code source". ### Installation ```bash go get github.com/koltyakov/gosip/cpass ``` ### Convertor ```go package main import ( "flag" "fmt" "github.com/koltyakov/gosip/cpass" ) func main() { var rawSecret string flag.StringVar(&rawSecret, "secret", "", "Raw secret string") flag.Parse() crypt := cpass.Cpass("") secret, _ := crypt.Encode(rawSecret) fmt.Println(secret) } ``` ### Encrypt secrets ```bash go run ./ -secret "MyP@s$word" #> -lywbAGD4iPYdJXDxLAQoMUbfBXBIQR2UZYl ``` When use result token/hash as a secret in `private.json` file(s). ### From sandbox Another option would be installing `cpass` from sandbox: ```bash go install github.com/koltyakov/gosip-sandbox/samples/cpass ``` And using `cpass` as a CLI, with no parameters the secret can be provided in a masked form without keeping it in console history: ```bash $ cpass Password to encode: ******** poXx8zaJM6gLazPCtv4rMVLoTuzX_1BvYJlMAQqK ``` # Compatibility matrix API methods compatibility When it comes to a code base which should support multiple platform versions, which APIs obviously changes with time and not aligned together, it can be a challenge maintaining-wise. Luckily, authentication methods are isolated and pretty static and HTTP client is generic for any SharePoint API consumption, no-matter REST, CSOM, or legacy SOAP are intended to be used. On the other side, the most of the help for a Go developer might come from Fluent API as it self-documented or intuitive usage-wise, covers mostly demanded use-cases, and abstracts most of the API nuances and complexity under the hood so a Go developer should not know lots of SharePoint since start. In most of the cases, when it come to a method which only supported let's say in SharePoint Online we put a special comment spotlighting versions support nuances. But a comment can be easily missed out. We plan to introduce sort of a tool for analysis unsupported methods in a custom code using Gosip Fluent API which verify that only supported methods are used for a targeted platform, this might happen in future in case of a reasonable demand. Until then we recommend manual verification of the methods and API entities support across the platform versions using [sp-metadata](https://github.com/koltyakov/sp-metadata) project. # Overview 💪 Contributing to Gosip client ### Intro First of all, thank you for considering contributing to the project! We really appreciate any activity around it. There are no small contributions and any investment can't be underestimated. You can contribute with feedback, finding and posting an issue, docs suggestion, code commit, or a star ⭐️. All of these encourage us supporting this and other our Open Source projects on a high level. Gosip is Open Source project hosted on GitHub. We, at [ARVO Systems](https://www.arvosys.com), and community, use the library on our production projects for our customers which makes it easier to guarantee stability and maintenance. The main project repo is located at . Contributing to the project follows a majority of GitHub and Open Source communities' principles. ### Code contributing guidance * Target your pull requests to the **dev** branch. * Add/update any docs articles related to your changes embeded to the code or as a separate notes to the PR, which we'd love to publish on docs site. * Include a test for any new functionality and ensure all existing tests are passing by running `go test` command(s), see [more here](/contributing/testing) * If you are fixing a bug, include a test that would have caught the bug you are fixing. * Keep your PRs as simple as possible and describe the changes to help the reviewer understand your work. * If you have an idea for a larger change to the library please [open an issue](https://github.com/koltyakov/gosip/issues) and let's discuss before you invest many hours - these are very welcome but want to ensure it is something we can merge before you spend the time. # Testing 🚦Gosip automated testing In a clone or fork. ## Authentication testing Create auth credentials store files in `./config` folder for corresponding strategies: * [private.onprem-adfs.json](/auth/strategies/adfs#on-premises-configuration) * [private.onprem-fba.json](/auth/strategies/fba#json) * [private.onprem-ntlm.json](/auth/strategies/ntlm#json) * [private.onprem-tmg.json](/auth/strategies/tmg#json) * [private.onprem-wap-adfs.json](/auth/strategies/adfs#on-premises-behing-wap-configuration) * [private.onprem-wap.json](/auth/strategies/adfs#on-premises-behing-wap-configuration) * [private.spo-addin.json](/auth/strategies/addin#json) * [private.spo-user.json](/auth/strategies/saml#json) * [private.spo-adfs.json](/auth/strategies/adfs#sharepoint-online-configuration) See [samples](https://github.com/koltyakov/gosip-docs/tree/3818da275a317ce807a5dbe14eade1086e60cc7f/contributing/config/samples/README.md). ```bash go test ./... -v -race -count=1 ``` Not provided auth configs and therefore strategies are ignored and not skipped in tests. ## API integration tests Create auth credentials store files in `./config/integration` folder for corresponding environments: * private.spo.json * private.2013.json // 2013 has its nuances with OData mod and not supported methods ```bash SPAUTH_ENVCODE=spo go test ./api/... -v -race -count=1 SPAUTH_ENVCODE=2013 go test ./api/... -v -race -count=1 ``` API integration tests are mostly targeted to SharePoint Online and not regularly processed on the legacy versions of the platform so you can face some test exceptions which still should be escaped with `t.Skip` and `envCode != "spo"` condition: ```go if envCode != "spo" { t.Skip("is not supported with old SharePoint versions") } ``` **Environment variables** * `SPAUTH_ENVCODE=code` environment variable switches target environments. `spo` is a default one. * `SPAPI_HEAVY_TESTS=true` turns on "heavy" methods, e.g. web creation. ## Run manual tests Modify `cmd/test/main.go` to include required scenarios and run: ```bash go run ./cmd/test ``` Optionally, you can provide a strategy to use with a corresponding flag: ```bash go run ./cmd/test -strategy adfs ``` ## Run CI tests Configure environment variables: * SPAUTH\_SITEURL * SPAUTH\_CLIENTID * SPAUTH\_CLIENTSECRET * SPAUTH\_USERNAME * SPAUTH\_PASSWORD ```bash go test ./... -v -race -count=1 SPAUTH_ENVCODE=spo SPAPI_HEAVY_TESTS=true go test ./api/... -v -race -count=1 ``` ## Test coverage Check [Codecov](https://codecov.io/gh/koltyakov/gosip). | Branch | Coverage | | ------ | ------------------------------------------------------------------------------- | | Master | ![codecov](https://codecov.io/gh/koltyakov/gosip/branch/master/graph/badge.svg) | | Dev | ![codecov](https://codecov.io/gh/koltyakov/gosip/branch/dev/graph/badge.svg) | We are targeted to keep code coverage higher than 80% for API and Auth methods altogether.