LogoLogo
GoDocGitHub
  • Introduction
  • FAQ
  • Authentication strategies
    • Overview
    • Strategies
      • Azure Certificate Auth
      • Azure Creds Auth
      • Azure Env-based Auth
      • Azure Device Flow
      • SAML Auth
      • AddIn Only
        • Configuration
      • NTLM Auth
      • NTLM (alternative)
      • On-Demand Auth
      • ADFS Auth
      • FBA Auth
      • TMG Auth
      • Anonymous
    • Dynamic auth
    • Custom Auth
  • SharePoint client
    • HTTP Client
    • Fluent API
    • Hooks
    • Retries
    • Context
  • Samples
    • Library Initiation
    • Basic CRUD
    • Documents
    • Chunk upload
    • Permissions
    • Groups & Users
    • Search API
    • User Profiles
    • Change API
    • Attachments
    • Record Management
    • Sending Emails
    • Property Bags
    • Recycle Bin
    • Feature management
    • Advanced item requests
    • Advanced add/update
    • Unmarshaling responses
  • Sandbox
    • Overview
  • Utilities
    • Headers presets
    • Cpass
    • Compatibility matrix
  • Contributing
    • Overview
    • Testing
Powered by GitBook
On this page
  • Get items by CAML
  • Render List Data
  • Render List Data as Stream
  • Pagination
  • Requesting large lists

Was this helpful?

Edit on GitHub
Export as PDF
  1. Samples

Advanced item requests

Advanced scenarios for getting list items

PreviousFeature managementNextAdvanced add/update

Last updated 5 years ago

Was this helpful?

There are so many nuances connected with requesting items in a list. By default, it's recommended using OData operations as the most simple, straightforward and RESTful approach.

But OData doesn't cover all of the needs and you got to switch back but not necessarily backward to methods. There a lot of gaps which can force you doing this: single valued MMD fields, group by requests, getting items from a view, to name just a few. Sometimes it's old known bugs or limitations, sometimes a specific functionality which was in the CAML days.

Get items by CAML

list := sp.Web().GetList("Lists/MyList")

caml := `
	<View>
		<Query>
			<Where>
				<Eq>
					<FieldRef Name='ID' />
					<Value Type='Number'>3</Value>
				</Eq>
			</Where>
		</Query>
	</View>
`

data, err := list.Items().GetByCAML(caml)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("%s\n", data)

Of course, CAML query can be slightly bit more complex than just this. 😁

For example, you can request data as in a list view:

list := sp.Web().GetList("Lists/MyList")
viewResp, err := list.Views().DefaultView().Get()
if err != nil {
	log.Fatal(err)
}

data, err := list.Items().GetByCAML(viewResp.Data().ListViewXML)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("%s\n", data)

However, when working with CAML there is more powerful methods.

Render List Data

list := sp.Web().GetList("Lists/MyList")

caml := `
	<View>
		<Query>
			<GroupBy Collapse="TRUE">
				<FieldRef Name="Competed" />
			</GroupBy>
		</Query>
	</View>
`

data, err := list.RenderListData(caml)
if err != nil {
	log.Fatal(err)
}

fmt.Printf("%s\n", data)

RenderListData* methods have completely different response structure. It's good and bad at the same time. The cons are that the results are not compatible and really different from OData methods, not only the shape but also values format. Sometimes it aches. The pros are that render list data methods provide something which is missed in OData responses: group by with collapsed data and only subtotals are possible, recurrent calendar events can be requested (yet it's now so shiny), to name a few.

RenderListData deals with GET request, which is a disadvantage as CAML query length is limited. This limitation is rarely can be met in practice, but the ridiculously complex conditions could fail due to this fact.

Render List Data as Stream

This method is super powerful, SharePoint Modern UI list views work using .RenderListDataAsStream. The method is the continuation of evolving of .RenderListData enhanced by the vendor for their practical needs of building the modern UI views.

The method deals with POST requests, almost have no length limitation, and operates with many-many options for covering all those aspects and features of the Modern UI view.

Pagination

Pagination in SharePoint lists is painful if you misses couple of moments. In a contrast to many databases queries where top and skip are straightforward thing to build an pagination, OData's $skiptoken is not what many think.

First of all, it's not a number of rows to skip before starting returning items on a next paged collection.

The simplest format of skip token is: Paged=TRUE&p_ID=5

The simplest reverse skip token is: Paged=TRUE&PagedPrev=TRUE&p_ID=5

Reverse token returns previous page content obviously.

By looking at p_ID part you'd think that item's ID is enough to construct a correct skip token, but it's not so. It's only correct for not sorted collections. If any $orderby modifier is applied, the amount of p_* parameters changes. Let's assume you sorted the list by Title, it will add something like p_Title=Smth, where "Smth" is the last row's Title value on current page collection.

Fortunately, skip tokens should not be constructed manually. REST return next page collection URI together with the responses for the current page collection.

In Gosip we have helper methods which makes it simpler working with pagination.

list := sp.Web().GetList("Lists/MyList")

page, err := list.Items().Select("Id").Top(100).GetPaged()
if err != nil {
	log.Fatal(err)
}

fmt.Printf("Page items %d\n", len(page.Items.Data()))

if page.HasNextPage() {
	nextPage, err := page.GetNextPage()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Next page items %d\n", len(nextPage.Items.Data()))
}

We're planning some improvements with pagination interfaces and scale the approach to all possible paged collections APIs but will try to introduce any backward incompatibilities as little as possible.

Requesting large lists

Large lists in SharePoint are those which amount of items is larger than view throttling limitation, the default limitation is 5000 items in a view. In On-Prem this value can be tweaked, in SPO this is a hard limit.

REST API can't return more than 5K items at once. Filter conditions based on indexing fields must be applied to trim down items number. It can be hard, though it's common for server-side processing requesting all items even if there are tens of thousands of items in a list. Such operations are not for immidiate actions but long running syncronizations.

In Gosip, .GetAll method is at disposal.

list := sp.Web().GetList("Lists/MyList")

allItems, err := list.Items().Select("Id,Title").Top(5000).GetAll()
if err != nil {
	log.Fatal(err)
}

// process allItems data

The method disables any ordering and filtering if applied, as ordering and filtering are not compatible with large lists.

Recommendations for getting all items from a large list:

  • Always specify only really required fields to retrieve in Select

  • Use Top equal to 5000 (as the default Top=100)

The method is in our plans to implement in Gosip Fluent API. But you always can craft that API consumption using AdHoc queries and .

Consider event-based synchronizations and partial

Consider logic

Basic CRUD
CAML
getting changes
search-based
HTTP Client